Summary
What you’ll impact
Our organization is seeking experienced engineers and scientists to develop evaluation infrastructure and systems that drive frontier LLM performance. The role involves designing frameworks to assess model improvements and ensure reliable, scalable production performance.
Responsibilities
What you'll do
- Build scalable, automated evaluation pipelines that integrate into model training and deployment workflows.
- Design, develop, and maintain robust evaluation frameworks and benchmarks for measuring LLM performance across diverse tasks and domains.
- Conduct rigorous statistical analysis of model outputs to identify failure modes, biases, and performance gaps.
- Partner with product and customer-facing teams to translate real-world use cases into meaningful evaluation criteria.
- Define and implement quantitative metrics that capture model quality, safety, reliability, and regression detection.
Requirements
What you’ll bring
- BS/MS/PhD in Computer Science, Machine Learning, Statistics, or a related field (or equivalent experience).
- At least 2 years of experience in ML evaluation, applied ML research, or a related engineering role.
- Experience with version control (Git), containerization (Docker), and cloud services (AWS, GCP, or Azure).
- Understanding of LLM fundamentals (autoregressive generation, instruction tuning, RLHF, in-context learning, decoding strategies).
- Proficiency in Python and ML frameworks such as PyTorch.
- Experience designing and implementing evaluation metrics and benchmarks for generative models.
- Excellent communication skills with the ability to distill complex evaluation results into actionable insights.