StateBench
by Ai2 (Allen Institute for AI)
What is StateBench? StateBench is a reasoning & knowledge eval made by Ai2 (Allen Institute for AI). A long-context state-tracking benchmark: it checks whether a model can keep track of how variables or entities change across a long document. It has four difficulty levels built from the same underlying sequence of assignments: values that never change, values that get overwritten, updates that build on the current value (like a running score) and assignments by reference such as swaps, where one missed step breaks the answer. The same problems come in several formats, such as code and spreadsheet edits. In Ai2's launch post (3 Nov 2026), frontier models handled the two easier levels but fell to 13% or lower on the longest hardest-level problems, and small recurrent and hybrid models trained directly on the task generalized much farther than transformers. Code and data are public on GitHub and Hugging Face.
Editor note
A long-context state-tracking benchmark: it checks whether a model can keep track of how variables or entities change across a long document. It has four difficulty levels built from the same underlying sequence of assignments: values that never change, values that get overwritten, updates that build on the current value (like a running score) and assignments by reference such as swaps, where one missed step breaks the answer. The same problems come in several formats, such as code and spreadsheet edits. In Ai2's launch post (3 Nov 2026), frontier models handled the two easier levels but fell to 13% or lower on the longest hardest-level problems, and small recurrent and hybrid models trained directly on the task generalized much farther than transformers. Code and data are public on GitHub and Hugging Face.
- What it tests
- Whether language models can follow chains of state changes over very long inputs, from simple overwrites to swaps and references.
- Original source
- github.com/allenai/state-bench
- Last updated
- Announced 2026-11-03 (repo last push 2026-10-23)
- License
- Apache-2.0
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
- MMLU — World knowledge and problem solving across many school and professional subjects.
Discussion
No one has discussed StateBench yet. Have you run it? Share what it caught, what it missed or how you set it up.