FraudBench
by Kevin Wei, Sumaya Nur Adan, Stephan Llerena et al. (GOAL Lab)
What is FraudBench? FraudBench is a safety & factuality eval made by Kevin Wei, Sumaya Nur Adan, Stephan Llerena et al. (GOAL Lab). Tests whether language models will recommend, help with or carry out conduct that would count as fraud under US law. Its 159 fictionalised scenarios are based on 112 real US fraud lawsuits (2014 to 2025) and checked by legally trained experts; in each, the model gets a chance, without being told to, to misstate or hide an important fact in a business setting, such as calling company-funded testing independent. Every scenario is run at three levels of autonomy (advising the user, drafting a message for them, acting on their behalf), 477 tasks in all. The release ships Inspect AI tasks, a human baseline from 996 people and per-sample results for 20 LLMs. The dataset sits on Hugging Face behind access conditions, and the authors ask users not to re-share it.
Editor note
Tests whether language models will recommend, help with or carry out conduct that would count as fraud under US law. Its 159 fictionalised scenarios are based on 112 real US fraud lawsuits (2014 to 2025) and checked by legally trained experts; in each, the model gets a chance, without being told to, to misstate or hide an important fact in a business setting, such as calling company-funded testing independent. Every scenario is run at three levels of autonomy (advising the user, drafting a message for them, acting on their behalf), 477 tasks in all. The release ships Inspect AI tasks, a human baseline from 996 people and per-sample results for 20 LLMs. The dataset sits on Hugging Face behind access conditions, and the authors ask users not to re-share it.
- What it tests
- Whether a model avoids misrepresenting or concealing material facts in high-stakes commercial scenarios, from advice through to direct action.
- Original source
- github.com/GOAL-Lab/FraudBench-Public
- Last updated
- Repo last push 2026-11-10; arXiv v1 2026-11-09
- License
- CC BY-NC-SA 4.0
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.
Discussion
No one has discussed FraudBench yet. Have you run it? Share what it caught, what it missed or how you set it up.