Summary
What you’ll impact
The LLM Evaluation Engineer evaluates new large language model releases and maintains the benchmark leaderboards that measure their real‑world performance. The role uses Python, Django, and AWS, employing tools like Docent to run and analyze benchmark suites across tasks such as law, finance, and coding. Responsibilities include integrating new models into the library, diagnosing error patterns, improving benchmarking infrastructure, and collaborating with communications to publish results. The engineer works in fast‑paced sprint cycles aligned with model releases, coordinating with model labs and cross‑functional teams to deliver high‑quality evaluations.
Responsibilities
What you'll do
- You'll be responsible for testing new models against our benchmarks as they're released; covering tasks in law, tax, coding, finance, social mobility, and more.
- You will analyze error modes of models, evaluate their strengths and weaknesses, and work with our communications team to release results.
- You will contribute directly to the leaderboards that make this possible.
- Evaluate new LLM model releases across the Vals AI suite of benchmarks
- Work directly with both open-source and closed-source foundation model labs in evaluating model performance
- Use tools like Docent to analyze common failure modes and patterns in model performance
- Work directly with our social media team to post interesting findings and results
- Add new models and maintain integrations in our model library
- Help improve and maintain the infrastructure we use to run benchmarks (agentic and non-agentic).
- This role follows the rhythm of model releases. Expect intense sprints in the days following a major launch, and calmer stretches in between releases.
Requirements
What you’ll bring
- Familiarity with the LLMs: You should already be familiar with the space - the current leading models, relative performance across them, how to use large language models in practice.
- Strong engineering fundamentals: You can build and ship quickly with high quality. You should have a track record of building things of significant scope (at jobs, side projects, open source, etc.)
- Python expertise: Significant experience in Python, especially in a professional setting.
- Team collaboration: Experience working in development sprints, Git workflows, and pull request reviews.
- Strong work ethic: Willingness to work long hours during model releases and get high-quality results out under tight deadlines.
- Location: We are an in-person team based in San Francisco. We will support your relocation or transportation as needed.