MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena)
MT-Bench or LMArena (Chatbot Arena)? MT-Bench is a fixed multi-turn question set you run yourself and grade with an LLM judge. LMArena ranks models from crowdsourced human votes on visitors' own prompts; you cannot run it on a private model.
| MT-Bench (LLM-as-a-judge paper) | LMArena (Chatbot Arena) | |
|---|---|---|
| Made by | LMSYS | LMArena (started by LMSYS) |
| Who judges | A strong LLM judge | Human visitors voting between two answers |
| Prompts | Fixed public question set | Visitors' own prompts |
| Run it yourself | Yes | No; rankings come from the public site |
| Measures | Multi-turn quality | Preference, not correctness |
| Watch out for | Judge biases (position, verbosity, self-enhancement) | Well-formatted wrong answers can win; prompts may not match your users |
| License | — | — |
Verdict: when to use which
Use MT-Bench (LLM-as-a-judge paper) if you need a repeatable test you can run on your own model.
Use LMArena (Chatbot Arena) if you want to see which public models people prefer on everyday prompts.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp