Writing & instruction following

XYBench

by Ai2, with Carnegie Mellon University, NVIDIA and Johns Hopkins University

What is XYBench? XYBench is a writing & instruction following eval made by Ai2, with Carnegie Mellon University, NVIDIA and Johns Hopkins University. Tests what an assistant does with an 'XY problem': a user asks how to make approach Y work when their real goal X needs a different approach (asking how to parse XML with regex when they just need the data). It has just over 8,000 such questions from Stack Overflow, Stack Exchange, WikiHow-based how-tos and a hand-curated set, each labelled with the stated request, the intended request and rubrics for a good answer. An LLM judge checks whether a free-form reply solves the stated request, solves the real goal, which one it emphasises, and whether it points out the misconception; there is also a multiple-choice setting. Ai2 reports that all five models it tested stayed more focused on the mistaken approach than human answers did. Data is on Hugging Face; code is on GitHub.

Page last updated 2026-10-18 · Evals Review

Editor note

Tests what an assistant does with an 'XY problem': a user asks how to make approach Y work when their real goal X needs a different approach (asking how to parse XML with regex when they just need the data). It has just over 8,000 such questions from Stack Overflow, Stack Exchange, WikiHow-based how-tos and a hand-curated set, each labelled with the stated request, the intended request and rubrics for a good answer. An LLM judge checks whether a free-form reply solves the stated request, solves the real goal, which one it emphasises, and whether it points out the misconception; there is also a multiple-choice setting. Ai2 reports that all five models it tested stayed more focused on the mistaken approach than human answers did. Data is on Hugging Face; code is on GitHub.

What it tests
Whether a model spots a user's misconception and steers the answer to their real goal instead of only the literal request.
Original source
github.com/allenai/XYBench
Last updated
Released 2026-11-13 (Ai2 blog); repo last push 2026-11-12; HF dataset last modified 2026-11-12; paper arXiv 2609.06842
License
Apache-2.0 (code); dataset CC BY-NC-SA 3.0 per Hugging Face

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related writing & instruction following evals

All Writing & instruction following evals

Discussion

No one has discussed XYBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions