FrontierPhysics
by Bingran You, Wenjun Ke, Ria Rosenauer et al. (BenchFlow, UC Berkeley and 35 other institutions)
What is FrontierPhysics? FrontierPhysics is an agents & tool use eval made by Bingran You, Wenjun Ke, Ria Rosenauer et al. (BenchFlow, UC Berkeley and 35 other institutions). An end-to-end physics research benchmark of 56 tasks built by 42 physicists from their own projects across nine domains. Agents get a PI-style brief, then must search the literature, plan, run simulations or data analysis, and deliver both checkable numbers and a paper manuscript. Grading combines deterministic verifiers with expert-written rubrics (blocking and scored criteria) applied by a reviewer agent. In the v0.1 paper, 22 models in 26 configurations were run three times per task; the best, Claude Opus 5.5 in Claude Code, passed 46 of 168 rollouts (27.4% pass@1), and failures traced mostly to weak rigour and expert intuition rather than missing physics knowledge.
Editor note
An end-to-end physics research benchmark of 56 tasks built by 42 physicists from their own projects across nine domains. Agents get a PI-style brief, then must search the literature, plan, run simulations or data analysis, and deliver both checkable numbers and a paper manuscript. Grading combines deterministic verifiers with expert-written rubrics (blocking and scored criteria) applied by a reviewer agent. In the v0.1 paper, 22 models in 26 configurations were run three times per task; the best, Claude Opus 5.5 in Claude Code, passed 46 of 168 rollouts (27.4% pass@1), and failures traced mostly to weak rigour and expert intuition rather than missing physics knowledge.
- What it tests
- Whether research agents can take an open physics question to a correct, publishable result with a manuscript.
- Original source
- arxiv.org/abs/2611.06910
- Last updated
- arXiv v1 2026-11-04
- License
- Not checked yet — see source
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed FrontierPhysics yet. Have you run it? Share what it caught, what it missed or how you set it up.