Accelerate Bench
by Bowen Liu, Yuhan Li, Nuo Chen et al.
What is Accelerate Bench? Accelerate Bench is a coding eval made by Bowen Liu, Yuhan Li, Nuo Chen et al.. 32 CPU optimisation tasks adapted from scientific applications (for example GROMACS, OpenFOAM and Quantum ESPRESSO) and supercomputing competitions. Instead of reproducing one known performance commit, the agent gets a whole working program, a fixed workload and CPU and memory limits, and eight hours to make it faster; a separate private verifier rebuilds the result, checks it on hidden inputs and times it against the original. The headline Score@1 averages correctness-gated runtime savings. On version 1.1 the paper (arXiv:2611.20882) reports 16 agents between 25.5% and 67.4%, led by Claude Sonnet 5.5 in Claude Code. Tasks ship as Harbor packages.
Editor note
32 CPU optimisation tasks adapted from scientific applications (for example GROMACS, OpenFOAM and Quantum ESPRESSO) and supercomputing competitions. Instead of reproducing one known performance commit, the agent gets a whole working program, a fixed workload and CPU and memory limits, and eight hours to make it faster; a separate private verifier rebuilds the result, checks it on hidden inputs and times it against the original. The headline Score@1 averages correctness-gated runtime savings. On version 1.1 the paper (arXiv:2611.20882) reports 16 agents between 25.5% and 67.4%, led by Claude Sonnet 5.5 in Claude Code. Tasks ship as Harbor packages.
- What it tests
- Application-level performance engineering by coding agents without breaking the required computation.
- Original source
- acceleratebench.com
- Last updated
- v1.1; arXiv:2611.20882 (16 Nov 2026); repo last push 2026-11-17 (GitHub API, read 18 Nov 2026)
- License
- Code Apache-2.0 (GitHub API, read 18 Nov 2026)
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed Accelerate Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.