MMLU vs MMLU-Pro
MMLU or MMLU-Pro? MMLU-Pro is the harder, more reasoning-heavy successor. It has ten answer options instead of four, drops easy and noisy MMLU questions, and lowers scores by 16 to 33 points. Use MMLU-Pro when MMLU no longer separates the models you are comparing.
| MMLU | MMLU-Pro | |
|---|---|---|
| Made by | Hendrycks et al. (UC Berkeley & collaborators) | TIGER-AI-Lab |
| Size | 57 subjects | 12,032 questions in 14 subject areas |
| Answer options | 4 (25% by chance) | 10 (10% by chance) |
| Question type | Mostly knowledge recall | More multi-step reasoning; added STEM and exam problems |
| Chain of thought | Direct answering often scores better | CoT can score about 20% higher than direct answering |
| Prompt sensitivity | 4–5% across 24 prompt styles | About 2% |
| Status | Top models score above 90% | Wider spread between strong models |
| License | MIT | Apache-2.0 |
Verdict: when to use which
Use MMLU if you need a broad, widely reported baseline or per-subject results for smaller models.
Use MMLU-Pro if you are comparing strong models whose MMLU scores sit within a few points of each other.
Dated numbers
- MMLU-Pro paper (2024): accuracy drops 16% to 33% compared with MMLU; prompt sensitivity falls from 4–5% to 2%. [MMLU-Pro paper (arXiv)]
- MMLU-Pro dataset card: GPT-4o 88.7% on MMLU vs 72.6% on MMLU-Pro; Claude-3-Opus 86.8% vs 68.5%; Llama-3-70B-Instruct 82.0% vs 56.2%. [MMLU-Pro dataset card (Hugging Face)]
FAQ
Why is MMLU-Pro harder than MMLU?
Ten options instead of four, fewer trivial or noisy questions, and more questions that need multi-step reasoning.
Are MMLU and MMLU-Pro scores comparable?
No. They are different question sets with different chance levels (25% vs 10%).
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · Terminal-Bench vs SWE-bench · GPQA vs Humanity's Last Exam · Ragas vs DeepEval