Writing & instruction following

IFBench

by Allen Institute for AI (Ai2) and University of Washington (Valentina Pyatkin, Saumya Malik, Victoria Graf et al.)

What is IFBench? IFBench is a writing & instruction following eval made by Allen Institute for AI (Ai2) and University of Washington (Valentina Pyatkin, Saumya Malik, Victoria Graf et al.). An instruction-following benchmark built to check whether models generalise to output constraints they were not tuned on, rather than to the small, familiar set that IFEval uses. It pairs 58 new out-of-domain constraints (counting, formatting, word and character manipulation, copying and similar) with held-out WildChat prompts, and every constraint has a Python function that checks the answer automatically, so no judge model is needed. An optional two-turn variant puts the task and the constraint in separate turns. The repository also ships the classic IFEval checkers and 29 extra training constraints for RL with verifiable rewards. The paper (arXiv:2507.02833, NeurIPS 2025 Datasets and Benchmarks) reports that leading models of the time, such as GPT-4.1 and Claude 3.7 Sonnet, scored below 50%.

Page last updated 2026-10-18 · Evals Review

Editor note

An instruction-following benchmark built to check whether models generalise to output constraints they were not tuned on, rather than to the small, familiar set that IFEval uses. It pairs 58 new out-of-domain constraints (counting, formatting, word and character manipulation, copying and similar) with held-out WildChat prompts, and every constraint has a Python function that checks the answer automatically, so no judge model is needed. An optional two-turn variant puts the task and the constraint in separate turns. The repository also ships the classic IFEval checkers and 29 extra training constraints for RL with verifiable rewards. The paper (arXiv:2507.02833, NeurIPS 2025 Datasets and Benchmarks) reports that leading models of the time, such as GPT-4.1 and Claude 3.7 Sonnet, scored below 50%.

What it tests
Precise instruction following on unseen, automatically verifiable output constraints, single-turn or split over two turns.
Original source
github.com/allenai/IFBench
Last updated
Repo allenai/IFBench last push 2026-11-13 (GitHub API, read 20 Nov 2026); HF dataset allenai/IFBench_test last modified 2026-11-02
License
Apache-2.0 (code, GitHub API, read 20 Nov 2026); HF test dataset shows no licence tag

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related writing & instruction following evals

All Writing & instruction following evals

Discussion

No one has discussed IFBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions