BIG-Bench Extra Hard (BBEH)
by Google DeepMind
What is BIG-Bench Extra Hard (BBEH)? BIG-Bench Extra Hard (BBEH) is a reasoning & knowledge eval made by Google DeepMind. A harder replacement for BIG-Bench Hard, which top models had nearly saturated: each BBH task is swapped for a new task that tests a similar reasoning skill at much higher difficulty. The full set has 4,520 examples and a 460-example mini version; in the original paper the best model listed, o3-mini (high), scored a harmonic mean of 44.8.
Editor note
A harder replacement for BIG-Bench Hard, which top models had nearly saturated: each BBH task is swapped for a new task that tests a similar reasoning skill at much higher difficulty. The full set has 4,520 examples and a 460-example mini version; in the original paper the best model listed, o3-mini (high), scored a harmonic mean of 44.8.
- What it tests
- Broad general reasoning beyond math and coding.
- Original source
- github.com/google-deepmind/bbeh
- Last updated
- Repo last push 2025-05-07
- License
- Apache-2.0
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
- MMLU — World knowledge and problem solving across many school and professional subjects.
Discussion
No one has discussed BIG-Bench Extra Hard (BBEH) yet. Have you run it? Share what it caught, what it missed or how you set it up.