Summary
What you’ll impact
The Senior Machine Learning Research Engineer is responsible for designing and developing novel benchmarks that evaluate real-world capabilities of large language models. The role leverages Python for research experimentation and collaborates with infrastructure teams to implement benchmark pipelines at scale on cloud platforms. Key duties include conducting rigorous experimental design, statistical analysis, and publishing findings to advance the evaluation community, while analyzing model performance across benchmarks. The engineer works closely with foundation model labs and enterprise partners, integrating feedback and ensuring benchmarks remain relevant to evolving AI capabilities.
Responsibilities
What you'll do
- Design and develop novel, high-impact benchmarks that assess challenging real-world capabilities
- Conduct research to ensure our benchmarks are valid, reliable, and meaningful
- Collaborate with foundation model labs and enterprises to understand evolving evaluation needs
- Analyze model performance across benchmarks and articulate results
- Publish research findings and contribute to the broader evaluation research community
- Work closely with the infrastructure team to implement your benchmark designs at scale
- Stay current with the latest developments in LLM capabilities and evaluation methodologies
Requirements
What you’ll bring
- Advanced research experience: Master's degree or PhD in Computer Science, NLP, Machine Learning, or related field. Undergrads with very strong research backgrounds may also be considered.
- Publication track record: Published papers in reputable venues (NeurIPS, ICML, ACL, EMNLP, etc.) with focus on NLP, ML evaluation, or benchmarking.
- Research methodology: Strong understanding of experimental design, statistical analysis, and evaluation frameworks.
- Technical skills: Proficiency in Python for research and experimentation.
- Communication: Ability to clearly communicate complex research ideas to both technical and non-technical audiences.
- Collaboration: Experience working in research teams and integrating feedback.
- Portfolio: Demonstrated track record of impactful research work.
- Location: We are an in-person team based in San Francisco. We will support your relocation or transportation as needed.