Safety & factuality

FraudBench

by Kevin Wei, Sumaya Nur Adan, Stephan Llerena et al. (GOAL Lab)

What is FraudBench? FraudBench is a safety & factuality eval made by Kevin Wei, Sumaya Nur Adan, Stephan Llerena et al. (GOAL Lab). Tests whether language models will recommend, help with or carry out conduct that would count as fraud under US law. Its 159 fictionalised scenarios are based on 112 real US fraud lawsuits (2014 to 2025) and checked by legally trained experts; in each, the model gets a chance, without being told to, to misstate or hide an important fact in a business setting, such as calling company-funded testing independent. Every scenario is run at three levels of autonomy (advising the user, drafting a message for them, acting on their behalf), 477 tasks in all. The release ships Inspect AI tasks, a human baseline from 996 people and per-sample results for 20 LLMs. The dataset sits on Hugging Face behind access conditions, and the authors ask users not to re-share it.

Page last updated 2026-10-18 · Evals Review

Editor note

Tests whether language models will recommend, help with or carry out conduct that would count as fraud under US law. Its 159 fictionalised scenarios are based on 112 real US fraud lawsuits (2014 to 2025) and checked by legally trained experts; in each, the model gets a chance, without being told to, to misstate or hide an important fact in a business setting, such as calling company-funded testing independent. Every scenario is run at three levels of autonomy (advising the user, drafting a message for them, acting on their behalf), 477 tasks in all. The release ships Inspect AI tasks, a human baseline from 996 people and per-sample results for 20 LLMs. The dataset sits on Hugging Face behind access conditions, and the authors ask users not to re-share it.

What it tests
Whether a model avoids misrepresenting or concealing material facts in high-stakes commercial scenarios, from advice through to direct action.
Original source
github.com/GOAL-Lab/FraudBench-Public
Last updated
Repo last push 2026-11-10; arXiv v1 2026-11-09
License
CC BY-NC-SA 4.0

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related safety & factuality evals

  • HarmBench — Robustness to jailbreaks and harmful requests.
  • TruthfulQA — Truthfulness versus imitating human misconceptions.
  • StrongREJECT — Real effectiveness of jailbreak attacks.
  • SimpleQA — Short-form factuality and calibration.

All Safety & factuality evals

Discussion

No one has discussed FraudBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions