XYBench
by Ai2, with Carnegie Mellon University, NVIDIA and Johns Hopkins University
What is XYBench? XYBench is a writing & instruction following eval made by Ai2, with Carnegie Mellon University, NVIDIA and Johns Hopkins University. Tests what an assistant does with an 'XY problem': a user asks how to make approach Y work when their real goal X needs a different approach (asking how to parse XML with regex when they just need the data). It has just over 8,000 such questions from Stack Overflow, Stack Exchange, WikiHow-based how-tos and a hand-curated set, each labelled with the stated request, the intended request and rubrics for a good answer. An LLM judge checks whether a free-form reply solves the stated request, solves the real goal, which one it emphasises, and whether it points out the misconception; there is also a multiple-choice setting. Ai2 reports that all five models it tested stayed more focused on the mistaken approach than human answers did. Data is on Hugging Face; code is on GitHub.
Editor note
Tests what an assistant does with an 'XY problem': a user asks how to make approach Y work when their real goal X needs a different approach (asking how to parse XML with regex when they just need the data). It has just over 8,000 such questions from Stack Overflow, Stack Exchange, WikiHow-based how-tos and a hand-curated set, each labelled with the stated request, the intended request and rubrics for a good answer. An LLM judge checks whether a free-form reply solves the stated request, solves the real goal, which one it emphasises, and whether it points out the misconception; there is also a multiple-choice setting. Ai2 reports that all five models it tested stayed more focused on the mistaken approach than human answers did. Data is on Hugging Face; code is on GitHub.
- What it tests
- Whether a model spots a user's misconception and steers the answer to their real goal instead of only the literal request.
- Original source
- github.com/allenai/XYBench
- Last updated
- Released 2026-11-13 (Ai2 blog); repo last push 2026-11-12; HF dataset last modified 2026-11-12; paper arXiv 2609.06842
- License
- Apache-2.0 (code); dataset CC BY-NC-SA 3.0 per Hugging Face
Related writing & instruction following evals
- EQ-Bench Creative Writing — Creative writing quality judged against a rubric.
- AlpacaEval — General helpfulness of chat responses.
- MT-Bench (LLM-as-a-judge paper) — Multi-turn conversation quality; how reliable LLM judges are.
- IFEval — Following explicit, verifiable instructions.
Discussion
No one has discussed XYBench yet. Have you run it? Share what it caught, what it missed or how you set it up.