MATH
by Hendrycks et al. (UC Berkeley)
What is MATH? MATH is a dataset of 12,500 competition mathematics problems from Dan Hendrycks and collaborators (NeurIPS 2021), each with a full step-by-step solution. Many model reports use MATH-500, a 500-problem subset.
Editor note
12,500 competition mathematics problems with full step-by-step solutions (NeurIPS 2021). Many reports use the 500-problem MATH-500 subset.
- What it tests
- Multi-step maths reasoning with exact answers.
- Original source
- github.com/hendrycks/math
- Last updated
- Repo last commit 2025-09-06
- License
- MIT
When to use MATH
- You want a widely reported maths reasoning score with exact answers.
- You want step-by-step reference solutions to train or check reasoning.
Limitations
- It is public and old, so contamination is a risk and top models score near the ceiling.
- Results differ between the full test set and subsets such as MATH-500; say which you used.
How to run it
- Download the dataset from the repository.
- Prompt the model, extract the final boxed answer and compare it with the reference.
Notable results
- NeurIPS 2021 paper: 12,500 challenging competition mathematics problems, each with a full step-by-step solution. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).
- MMLU — World knowledge and problem solving across many school and professional subjects.