Nous Research
Full TimeDesign and maintain evaluation infrastructure for AI agents, building judge calibration protocols, extending benchmarks like GAIA and SWE-bench, and running failure analysis on model outputs. You'll need 3+ years in software engineering, ML engineering, or data science with concrete evaluation experience. Strong Python, Git, Docker, and Linux skills are essential, along with hands-on LLM experience and understanding of eval statistics like Cohen's κ. At least three of these matter: LLM-as-judge calibration knowledge, failure analysis expertise, familiarity with multiple agent benchmarks, eval dataset design, or non-determinism handling.
Written from this posting by Neural Jobs AI. The full description is below.
The Role
You'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one.
Responsibilities
Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task
Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it
Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA
Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes
Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use
Qualifications
3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job
Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short
Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents
Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it
Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)
Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric
At least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance
You communicate clearly to both researchers and engineers, in the right language for each
You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help
Preferred
RLVR / RLHF pipeline experience
Training data curation experience
Distributed eval orchestration experience
Benchmark design from scratch
Red teaming and adversarial eval experience
Familiarity with psychometrics or measurement theory
Sign in and we will show how your role, experience, salary and location line up against what this employer asked for.
Check my match
Americas (US time zones Remote
Search by role, company, or anything a posting mentions.
