AlpacaEval vs MT-Bench (LLM-as-a-judge paper)
AlpacaEval or MT-Bench? AlpacaEval compares a model's single-turn answers with a baseline's using an LLM judge, and its length-controlled win rate correlates 0.98 with Chatbot Arena. MT-Bench grades multi-turn conversations with a strong LLM judge and comes from the paper that studied judge biases.
| AlpacaEval | MT-Bench (LLM-as-a-judge paper) | |
|---|---|---|
| Made by | Stanford Tatsu Lab | LMSYS |
| Turns | Single turn | Multi-turn |
| Scoring | Win rate against a baseline, length-controlled by default | LLM judge grades responses |
| Agreement with humans | Spearman 0.98 with Chatbot Arena (length-controlled) | GPT-4 judge over 80% agreement with human preferences |
| Watch out for | Judges favour longer answers; report judge and baseline | Fixed public set can be overfit; judge biases apply |
| License | Apache-2.0 | — |
Verdict: when to use which
Use AlpacaEval if you want a fast, cheap ranking of instruction-following chat models.
Use MT-Bench (LLM-as-a-judge paper) if multi-turn quality matters or you are studying LLM-as-judge reliability.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp