Agent-as-a-Judge (DevAI)
by Meta AI & KAUST
What is Agent-as-a-Judge (DevAI)? Agent-as-a-Judge is an evaluation method in which an agentic system, one that can build a graph of a project, locate and read files and retrieve parts of a trajectory, judges another agent's work requirement by requirement. The paper (Meta AI and KAUST) introduces it with DevAI, a benchmark of 55 AI-development tasks and 365 hierarchical requirements.
Editor note
A method that uses an agent, which can read files, locate code and inspect trajectories, to judge other agents requirement by requirement, plus DevAI: 55 AI-development tasks with 365 hierarchical requirements. In the paper it agreed with human consensus about 90% of the time versus about 70% for LLM-as-a-judge.
- What it tests
- Coding agents on full AI-development tasks, judged step by step instead of pass/fail.
- Original source
- arxiv.org/abs/2410.10934
- Last updated
- Repo last commit 2026-03-28
- License
- MIT
When to use Agent-as-a-Judge (DevAI)
- You evaluate agents whose output is a whole workspace (code, files, results), where a single pass/fail misses where they went wrong.
- You want intermediate feedback per requirement, not only a final resolve rate.
- Human review is too slow: in the paper the agent judge cost $30.58 and 118 minutes versus about $1,297 and 86.5 hours for three human experts.
Limitations
- The evidence is one proof of concept on 55 code-generation tasks judging three agent frameworks; results may not carry over to other domains.
- The judge needs good context: in ablations, adding planning, memory or search modules did not help or hurt alignment.
How to run it
- Get the DevAI tasks from Hugging Face (devai-benchmark) and the judge code from the official repository.
- Run your developer agent on a task and keep its workspace (and trajectory, for gray-box judging).
- Run the Agent-as-a-Judge script to mark each requirement satisfied or not.
Notable results
- Agent-as-a-Judge agreed with the human consensus on about 90% of requirements, versus about 70% for LLM-as-a-judge (e.g. judging OpenHands: 90.44% vs 60.38% black-box). [Agent-as-a-Judge paper (arXiv)]
- Human judges: the best agents (GPT-Pilot, OpenHands) met about 29% of DevAI requirements with dependencies counted and fully solved only 1 of 55 tasks. [Agent-as-a-Judge paper (arXiv)]
Sources
FAQ
What is Agent-as-a-Judge?
Using an agent with tools, such as reading files and searching a project, to evaluate another agent's work step by step. It extends LLM-as-a-judge, where a single LLM call scores an output.
What is DevAI?
The benchmark released with the paper: 55 realistic AI-development tasks with 365 hierarchical requirements and 125 softer preferences.
Compare Agent-as-a-Judge (DevAI)
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.