IFBench
by Allen Institute for AI (Ai2) and University of Washington (Valentina Pyatkin, Saumya Malik, Victoria Graf et al.)
What is IFBench? IFBench is a writing & instruction following eval made by Allen Institute for AI (Ai2) and University of Washington (Valentina Pyatkin, Saumya Malik, Victoria Graf et al.). An instruction-following benchmark built to check whether models generalise to output constraints they were not tuned on, rather than to the small, familiar set that IFEval uses. It pairs 58 new out-of-domain constraints (counting, formatting, word and character manipulation, copying and similar) with held-out WildChat prompts, and every constraint has a Python function that checks the answer automatically, so no judge model is needed. An optional two-turn variant puts the task and the constraint in separate turns. The repository also ships the classic IFEval checkers and 29 extra training constraints for RL with verifiable rewards. The paper (arXiv:2507.02833, NeurIPS 2025 Datasets and Benchmarks) reports that leading models of the time, such as GPT-4.1 and Claude 3.7 Sonnet, scored below 50%.
Editor note
An instruction-following benchmark built to check whether models generalise to output constraints they were not tuned on, rather than to the small, familiar set that IFEval uses. It pairs 58 new out-of-domain constraints (counting, formatting, word and character manipulation, copying and similar) with held-out WildChat prompts, and every constraint has a Python function that checks the answer automatically, so no judge model is needed. An optional two-turn variant puts the task and the constraint in separate turns. The repository also ships the classic IFEval checkers and 29 extra training constraints for RL with verifiable rewards. The paper (arXiv:2507.02833, NeurIPS 2025 Datasets and Benchmarks) reports that leading models of the time, such as GPT-4.1 and Claude 3.7 Sonnet, scored below 50%.
- What it tests
- Precise instruction following on unseen, automatically verifiable output constraints, single-turn or split over two turns.
- Original source
- github.com/allenai/IFBench
- Last updated
- Repo allenai/IFBench last push 2026-11-13 (GitHub API, read 20 Nov 2026); HF dataset allenai/IFBench_test last modified 2026-11-02
- License
- Apache-2.0 (code, GitHub API, read 20 Nov 2026); HF test dataset shows no licence tag
Related writing & instruction following evals
- EQ-Bench Creative Writing — Creative writing quality judged against a rubric.
- AlpacaEval — General helpfulness of chat responses.
- MT-Bench (LLM-as-a-judge paper) — Multi-turn conversation quality; how reliable LLM judges are.
- IFEval — Following explicit, verifiable instructions.
Discussion
No one has discussed IFBench yet. Have you run it? Share what it caught, what it missed or how you set it up.