FabScore
by Hui Chen, James Xu Zhao, Dongfu Jiang et al. (National University of Singapore with Waterloo, Google, UC Merced, UC Davis and University of Washington)
What is FabScore? FabScore is a rubric libraries & frameworks eval made by Hui Chen, James Xu Zhao, Dongfu Jiang et al. (National University of Singapore with Waterloo, Google, UC Merced, UC Davis and University of Washington). An evaluation framework that checks AI-generated research papers against their own code for made-up results. It pulls every number and figure label out of a paper as a separate claim, then a coding agent reads and runs the accompanying code and gives each claim one of six verdicts: data, experiment or result fabrication, verified, or unverifiable (no code found, or not enough evidence). The authors ran it on 144 papers with code from AI Scientist, MLR-Agent, Agents4Science and FARS: 21.2% of 6,978 extracted claims were fabricated, mostly experiment fabrication, and 70.4% of the 54 real Agents4Science submissions contained at least one. The paper reports 98.6% precision against human checks and says AI reviewers missed over 85% of the fabrications it found. It was presented at the ICML 2026 AI4Science Workshop.
Editor note
An evaluation framework that checks AI-generated research papers against their own code for made-up results. It pulls every number and figure label out of a paper as a separate claim, then a coding agent reads and runs the accompanying code and gives each claim one of six verdicts: data, experiment or result fabrication, verified, or unverifiable (no code found, or not enough evidence). The authors ran it on 144 papers with code from AI Scientist, MLR-Agent, Agents4Science and FARS: 21.2% of 6,978 extracted claims were fabricated, mostly experiment fabrication, and 70.4% of the 54 real Agents4Science submissions contained at least one. The paper reports 98.6% precision against human checks and says AI reviewers missed over 85% of the fabrications it found. It was presented at the ICML 2026 AI4Science Workshop.
- What it tests
- Whether papers written by AI research agents report data, experiments and results that their own code actually supports.
- Original source
- github.com/chchenhui/fabscore
- Last updated
- Repo last push 2026-09-08
- License
- MIT
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.
Discussion
No one has discussed FabScore yet. Have you run it? Share what it caught, what it missed or how you set it up.