MMLU
by Hendrycks et al. (UC Berkeley & collaborators)
What is MMLU? MMLU is a reasoning & knowledge eval made by Hendrycks et al. (UC Berkeley & collaborators). Multiple-choice questions across 57 subjects, from elementary mathematics and US history to computer science and law, testing broad academic and professional knowledge.
Editor note
Multiple-choice questions across 57 subjects, from elementary mathematics and US history to computer science and law, testing broad academic and professional knowledge.
- What it tests
- World knowledge and problem solving across many school and professional subjects.
- Original source
- github.com/hendrycks/test
- Last updated
- Repo last commit 2023-04-24
- License
- MIT
When to use MMLU
- You want a widely reported baseline for general knowledge across 57 subjects, so you can compare with many published results.
- You want per-subject results: the original leaderboard reports Humanities, Social Sciences, STEM and Other separately.
- You need a harder version: MMLU-Pro adds reasoning-focused questions, raises the options from four to ten and removes trivial and noisy questions.
Limitations
- It is multiple choice, so it measures recognition of the right answer, not free-form explanation.
- Scores at the top have levelled off. The MMLU-Pro authors note that model performance on MMLU has begun to plateau, which makes differences hard to see, and the HLE paper reports that LLMs now score over 90%.
- Results change with prompting: the MMLU-Pro paper measured 4–5% score sensitivity to prompt style on MMLU (about 2% on MMLU-Pro).
How to run it
- Download the test from the original repository (data.tar) or the Hugging Face copy (cais/mmlu).
- Run it in an eval harness. Stanford's HELM, for example, includes MMLU as a run entry. Report few-shot or zero-shot and the prompt format.
pip install crfm-helm
helm-run --run-entries mmlu:subject=philosophy,model=<model> --suite my-suite --max-eval-instances 10Notable results
- In the original paper, most models were near random chance; the largest GPT-3 model was almost 20 points above chance on average, and results were weakest on subjects such as morality and law. [Paper (arXiv)]
- On MMLU-Pro, accuracy dropped by 16% to 33% compared with MMLU, and chain-of-thought helped more than on MMLU. [MMLU-Pro paper (arXiv)]
Sources
Compare MMLU
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).