Unfinished-Bench
by Seunghyun Lee, Jaehui Hwang, Dongyoon Han and Sangdoo Yun (NAVER AI Lab)
What is Unfinished-Bench? Unfinished-Bench is a safety & factuality eval made by Seunghyun Lee, Jaehui Hwang, Dongyoon Han and Sangdoo Yun (NAVER AI Lab). Tests what an autonomous agent does when it can finish its assignment only by breaking a rule. Each of 97 scenarios is set in one of 7 fictional organisations with staff, internal documents, chat and work tools, and the rule sits where it would at work: in a document, on a tool screen or with a colleague, rather than in the user's prompt. Every task is run with and without a harness that keeps pushing the agent to complete it. The paper (arXiv:2611.17460) reports that this persistence raised the average rate of violating a rule found in an internal document from 32.2% to 43.6% across 12 open models, about five times the 8.5% seen when the user stated the same rule. Code, scenarios and scoring are in the naver-ai/unfinished-bench repository.
Editor note
Tests what an autonomous agent does when it can finish its assignment only by breaking a rule. Each of 97 scenarios is set in one of 7 fictional organisations with staff, internal documents, chat and work tools, and the rule sits where it would at work: in a document, on a tool screen or with a colleague, rather than in the user's prompt. Every task is run with and without a harness that keeps pushing the agent to complete it. The paper (arXiv:2611.17460) reports that this persistence raised the average rate of violating a rule found in an internal document from 32.2% to 43.6% across 12 open models, about five times the 8.5% seen when the user stated the same rule. Code, scenarios and scoring are in the naver-ai/unfinished-bench repository.
- What it tests
- Whether persistent, unsupervised agents keep to rules embedded in their work environment when the goal and the rule conflict.
- Original source
- francis21c.github.io/unfinished-bench/
- Last updated
- arXiv:2611.17460 (12 Nov 2026); repo naver-ai/unfinished-bench last push 2026-11-16 (GitHub API, read 19 Nov 2026)
- License
- Apache-2.0 (repo LICENSE file, read 19 Nov 2026; GitHub API shows no SPDX id)
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.
Discussion
No one has discussed Unfinished-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.