Summary
What you’ll impact
The Head of AI Evaluation Research defines and advances methodologies to assess large language models and related AI systems. The role leverages Python and AWS cloud services to build benchmark frameworks and evaluation pipelines. Responsibilities include creating novel evaluation paradigms, publishing influential research, and shaping the research portfolio with enterprise customers and partner labs. Additionally, the leader recruits and mentors a research team while aligning efforts with founders and product stakeholders.
Responsibilities
What you'll do
- Advance the science of evaluation. The methodologies the field uses today — judge models, human-in-the-loop, static benchmarks — were built for a previous generation of models and break down on long-horizon, real-world tasks. You'll develop the new paradigms.
- Oversee Vals' broader research portfolio, setting direction across the projects already underway and the ones we haven't started yet.
- Publish work that moves the field forward. We want Vals' research to be cited, not just shipped.
- Recruit and grow a research team alongside the founders.
- Work directly with our enterprise customers and lab partners on the evaluation problems they actually have.
Requirements
What you’ll bring
- A PhD in ML/NLP (in progress or completed), or equivalent industry research track record
- Deep familiarity with the LLM evaluation landscape: existing benchmarks, their failure modes, judge-model approaches, human-in-the-loop methodologies.
- A bias toward research that affects what people actually deploy, rather than benchmarks that are easy to game.
- Strong written and verbal communication. You'll publish, present, and talk to customers and labs.
- Ability to work in-person, in San Francisco.