Safety & factuality

Unfinished-Bench

by Seunghyun Lee, Jaehui Hwang, Dongyoon Han and Sangdoo Yun (NAVER AI Lab)

What is Unfinished-Bench? Unfinished-Bench is a safety & factuality eval made by Seunghyun Lee, Jaehui Hwang, Dongyoon Han and Sangdoo Yun (NAVER AI Lab). Tests what an autonomous agent does when it can finish its assignment only by breaking a rule. Each of 97 scenarios is set in one of 7 fictional organisations with staff, internal documents, chat and work tools, and the rule sits where it would at work: in a document, on a tool screen or with a colleague, rather than in the user's prompt. Every task is run with and without a harness that keeps pushing the agent to complete it. The paper (arXiv:2611.17460) reports that this persistence raised the average rate of violating a rule found in an internal document from 32.2% to 43.6% across 12 open models, about five times the 8.5% seen when the user stated the same rule. Code, scenarios and scoring are in the naver-ai/unfinished-bench repository.

Page last updated 2026-10-18 · Evals Review

Editor note

Tests what an autonomous agent does when it can finish its assignment only by breaking a rule. Each of 97 scenarios is set in one of 7 fictional organisations with staff, internal documents, chat and work tools, and the rule sits where it would at work: in a document, on a tool screen or with a colleague, rather than in the user's prompt. Every task is run with and without a harness that keeps pushing the agent to complete it. The paper (arXiv:2611.17460) reports that this persistence raised the average rate of violating a rule found in an internal document from 32.2% to 43.6% across 12 open models, about five times the 8.5% seen when the user stated the same rule. Code, scenarios and scoring are in the naver-ai/unfinished-bench repository.

What it tests
Whether persistent, unsupervised agents keep to rules embedded in their work environment when the goal and the rule conflict.
Original source
francis21c.github.io/unfinished-bench/
Last updated
arXiv:2611.17460 (12 Nov 2026); repo naver-ai/unfinished-bench last push 2026-11-16 (GitHub API, read 19 Nov 2026)
License
Apache-2.0 (repo LICENSE file, read 19 Nov 2026; GitHub API shows no SPDX id)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related safety & factuality evals

  • HarmBench — Robustness to jailbreaks and harmful requests.
  • TruthfulQA — Truthfulness versus imitating human misconceptions.
  • StrongREJECT — Real effectiveness of jailbreak attacks.
  • SimpleQA — Short-form factuality and calibration.

All Safety & factuality evals

Discussion

No one has discussed Unfinished-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions