RE-Bench
by METR
What is RE-Bench? RE-Bench (Research Engineering Benchmark, v1) is METR's set of 7 open-ended ML research-engineering environments, released with data from 71 eight-hour attempts by 61 human experts so agents can be compared directly with people.
Editor note
7 open-ended ML research-engineering environments, with data from 71 eight-hour attempts by 61 human experts. With 2 hours per environment the best agents scored 4x higher than the experts, but people gained more from extra time and reached 2x the top agent's score at 32 hours.
- What it tests
- Agents on AI research and engineering work, compared with human experts.
- Original source
- github.com/METR/RE-Bench
- Last updated
- Repo last commit 2025-10-16
- License
- MIT
When to use RE-Bench
- You assess how far agents can automate AI research and development, a capability frontier safety policies track.
- You want a human baseline measured on the same tasks and time budgets.
Limitations
- Seven environments is a small sample.
- METR asks users to keep tasks out of training data and not publish unprotected solutions; some files sit in password-protected archives.
How to run it
- Clone the repository; tasks follow the METR Task Standard.
- Follow its setup guide to run them with Vivaria and METR's open-source agent scaffolding.
Notable results
- Paper: at a 2-hour budget per environment the best agents scored 4x higher than human experts; at 32 total hours humans reached 2x the top agent's score. 82% of expert attempts scored above zero. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.