BIG-Bench Hard (BBH)
by Suzgun et al. (from Google's BIG-Bench)
What is BIG-Bench Hard (BBH)? BIG-Bench Hard (BBH) is a suite of 23 tasks from Google's BIG-Bench on which earlier language models did not beat the average human rater. Its paper showed that chain-of-thought prompting closes much of that gap.
Editor note
23 BIG-Bench tasks on which earlier language models did worse than the average human rater. The paper showed chain-of-thought prompting let Codex pass the human average on 17 of the 23, so scores depend heavily on the prompting method.
- What it tests
- Multi-step reasoning across varied puzzle-style tasks.
- Original source
- github.com/suzgunmirac/BIG-Bench-Hard
- Last updated
- Repo last commit 2024-06-25
- License
- MIT
When to use BIG-Bench Hard (BBH)
- You want a widely reported multi-step reasoning score for base or smaller models.
- You want to compare prompting methods: the repo ships chain-of-thought prompts for every task.
Limitations
- Frontier models now score high, so it separates them less than newer sets.
- Scores swing with prompting (few-shot vs chain-of-thought); report the method.
How to run it
- Get the task files from the repository's bbh folder and prompts from cot-prompts.
- Run the model with the few-shot or chain-of-thought prompts and score exact match against the task answers.
Notable results
- Paper: with chain-of-thought prompting, PaLM beat the average human rater on 10 of 23 tasks and Codex on 17 of 23. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).
- MMLU — World knowledge and problem solving across many school and professional subjects.