GPQA vs Humanity's Last Exam
GPQA or Humanity's Last Exam? GPQA is a small, expert-written multiple-choice test in three sciences (448 questions; Diamond subset 198). Humanity's Last Exam is larger and broader: 2,500 questions across dozens of subjects, some with images, in multiple-choice or short-answer form. Use HLE when GPQA no longer separates the models you compare.
| GPQA | Humanity's Last Exam | |
|---|---|---|
| Made by | NYU & collaborators | Center for AI Safety & Scale AI |
| Size | 448 questions; Diamond subset 198 | 2,500 public questions plus a private held-out set |
| Subjects | Biology, physics, chemistry | Dozens of subjects, with a large share of mathematics |
| Format | Multiple choice (4 options) | Multiple choice and short exact answer; some with images |
| Grading | Answer key | Official scripts use a separate judge step |
| Human baseline | Experts 65%, skilled non-experts with web 34% | Questions written at the edge of expert knowledge |
| Published | COLM 2024 | Nature 2025 |
| License | MIT | MIT |
Verdict: when to use which
Use GPQA if you want a cheap, focused science-reasoning check that most model cards report (usually Diamond).
Use Humanity's Last Exam if you need a harder, broader academic test for frontier models. Note whether tools were enabled.
Dated numbers
- GPQA paper: PhD-level experts 65% (74% after discounting mistakes they later spotted), skilled non-experts with web access 34%, best GPT-4 baseline 39%. [GPQA paper (arXiv)]
- HLE paper (Nature, 2025): 2,500 questions; the leading LLMs tested showed low accuracy and poor calibration. [HLE paper (Nature)]
FAQ
Is HLE harder than GPQA?
Yes, by design. It was built after benchmarks such as MMLU passed 90% and covers far more subjects.
Should I report GPQA or GPQA Diamond?
Most model releases report Diamond (198 questions); say which subset you used.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · Terminal-Bench vs SWE-bench · MMLU vs MMLU-Pro · Ragas vs DeepEval