LLM-as-a-Judge vs Agent-as-a-Judge
LLM-as-a-Judge or Agent-as-a-Judge? LLM-as-a-judge asks one model to score an output from the text it is shown; it works well for chat answers, where GPT-4 matched human preferences over 80% of the time. Agent-as-a-judge gives the judge tools to explore a whole workspace and checks each requirement; on agent coding tasks it matched human consensus about 90% of the time versus about 70% for LLM-as-a-judge. Use the agent judge when the output is a project, not a reply.
| LLM-as-a-Judge | Agent-as-a-Judge | |
|---|---|---|
| Made by | LMSYS | Meta AI & KAUST |
| Judge | One LLM call over the shown text | An agent that can map the project, locate and read files, and retrieve trajectory steps |
| Best fit | Chat answers, single outputs | Agent workspaces with many requirements |
| Granularity | Score or pairwise preference | Each requirement satisfied or not |
| Agreement with humans | Over 80% with human preferences on MT-Bench (GPT-4) | About 90% with human consensus on DevAI, vs about 70% for LLM-as-a-judge |
| Known issues | Position, verbosity and self-enhancement biases | Proof of concept on 55 code tasks; planning and memory modules did not help |
| Cost (DevAI) | $29.63, 11 minutes | $30.58, 118 minutes (humans: ~$1,297, 86.5 hours) |
| License | — | MIT |
Verdict: when to use which
Use MT-Bench (LLM-as-a-judge paper) if you grade single responses at scale and can check the judge against some human labels.
Use Agent-as-a-Judge if you grade agents that produce files and code, and need to know which requirement failed.
Dated numbers
- MT-Bench paper (2023): strong LLM judges such as GPT-4 matched controlled and crowdsourced human preferences over 80% of the time, about the level humans agree with each other. [Judging LLM-as-a-Judge (MT-Bench paper, arXiv)]
- Agent-as-a-Judge paper: judging OpenHands, agent judge 90.44% alignment with humans vs 60.38% for LLM-as-a-judge (black-box). [Agent-as-a-Judge paper (arXiv)]
FAQ
Is Agent-as-a-Judge just LLM-as-a-Judge with tools?
Roughly, yes. The paper calls it an extension of LLM-as-a-judge that adds agentic features, so it can give feedback on the whole task-solving process.
Which is cheaper?
In the DevAI study the costs were similar ($30.58 vs $29.63), but the agent judge took about ten times longer to run.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · Terminal-Bench vs SWE-bench · MMLU vs MMLU-Pro · GPQA vs Humanity's Last Exam · Ragas vs DeepEval · Terminal-Bench 2.1 vs Terminal-Bench 4.0 · ARC-AGI-2 vs ARC-AGI-3 · LiveCodeBench vs SWE-bench · DeepEval vs Promptfoo