MMLU vs GPQA
MMLU or GPQA? MMLU is a broad multiple-choice exam across 57 subjects where top LLMs now score over 90%. GPQA is a small set of 448 expert-written biology, physics and chemistry questions built so web search does not help much.
| MMLU | GPQA | |
|---|---|---|
| Made by | Hendrycks et al. (UC Berkeley & collaborators) | NYU & collaborators |
| Scope | 57 subjects | Biology, physics, chemistry |
| Size | Large, broad | 448 questions in the main set |
| Format | Multiple choice (4 options) | Multiple choice |
| Difficulty | Top scores have plateaued above 90% | Experts 65%, skilled non-experts with web 34%, best GPT-4 baseline 39% in the paper |
| Watch out for | 4–5% sensitivity to prompt style | Small, so a few questions move the score; don't repost questions |
| License | MIT | MIT |
Verdict: when to use which
Use MMLU if you want a broad, widely reported knowledge baseline with per-subject results.
Use GPQA if you need hard expert science questions that still separate strong models.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp