TwinLab-Bench
by Yang Wan and Linchao Zhu (ZJUSCL)
What is TwinLab-Bench? TwinLab-Bench is an agents & tool use eval made by Yang Wan and Linchao Zhu (ZJUSCL). Tests AI-scientist agents on experiments they design themselves. Each of the 52 tasks is rebuilt from a real laboratory's records (20 physical sciences, 20 life sciences, 12 engineering): the agent gets the notebooks, raw data and interim conclusions the lab had at one point, requests experiments within a budget, and writes a report that is compared with a hidden answer taken from what the lab found later. A per-task environment agent returns the lab's recorded result when the experiment was actually run and otherwise uses calculators built from the lab's records; closed-book probes check for answers recalled from memory. The authors report the best model tested averaging 45 out of 100 (arXiv:2611.26248).
Editor note
Tests AI-scientist agents on experiments they design themselves. Each of the 52 tasks is rebuilt from a real laboratory's records (20 physical sciences, 20 life sciences, 12 engineering): the agent gets the notebooks, raw data and interim conclusions the lab had at one point, requests experiments within a budget, and writes a report that is compared with a hidden answer taken from what the lab found later. A per-task environment agent returns the lab's recorded result when the experiment was actually run and otherwise uses calculators built from the lab's records; closed-book probes check for answers recalled from memory. The authors report the best model tested averaging 45 out of 100 (arXiv:2611.26248).
- What it tests
- Open-ended experimental design and reporting by AI-scientist agents, grounded in real lab records.
- Original source
- twinlab-bench.github.io/
- Last updated
- arXiv:2611.26248 (19 Nov 2026); repo last pushed 2026-11-20 (GitHub API) and HF dataset ZJUSCL/TwinLab-Bench updated 2026-11-20 (read 22 Nov 2026)
- License
- Not checked yet — see source
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed TwinLab-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.