GSM8K
by OpenAI
What is GSM8K? GSM8K is OpenAI's dataset of 8.5K grade-school maths word problems written by human problem writers, split into 7.5K training and 1K test problems. Each needs 2 to 8 steps of basic arithmetic.
Editor note
8.5K grade-school maths word problems written by people: 7.5K for training and 1K for testing. Frontier models now score near the ceiling.
- What it tests
- Basic multi-step arithmetic reasoning.
- Original source
- github.com/openai/grade-school-math
- Last updated
- Repo last commit 2024-01-21
- License
- Not checked yet — see source
When to use GSM8K
- You test small or mid-sized models on multi-step arithmetic reasoning.
- You need a cheap, quick sanity check.
Limitations
- Frontier models are near the ceiling, so it no longer separates them.
- Its test set is public and widely copied, so contamination is likely.
How to run it
- Load the test split from the repository or Hugging Face (openai/gsm8k).
- Score exact match on the final number; lm-evaluation-harness has a ready gsm8k task.
Notable results
- README: 8.5K problems by human writers, 7.5K train and 1K test. [Official repository]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).
- MMLU — World knowledge and problem solving across many school and professional subjects.