FrontierCS 2
by FrontierCS team (Qiuyang Mang, Wenhao Chai et al.; 61 authors)
What is FrontierCS 2? FrontierCS 2 is a coding eval made by FrontierCS team (Qiuyang Mang, Wenhao Chai et al.; 61 authors). A research-and-development benchmark for coding agents: 48 released tasks in 13 areas such as databases, compilers, GPU kernels, high-performance computing, security and robotics. Each task starts from a published paper's own code with the paper's key contribution removed; the agent gets five hours in the repository, can request scores on development workloads, and passes only if its final patch matches or beats the paper's method on every metric of hidden held-out workloads. The paper (arXiv:2610.29932) reports 12 agents, with Claude Opus 5.5 passing 58% of its runs and 16 tasks passed by no run. More than 100 further tasks are kept private for later release, and results are mirrored on a Kaggle leaderboard.
Editor note
A research-and-development benchmark for coding agents: 48 released tasks in 13 areas such as databases, compilers, GPU kernels, high-performance computing, security and robotics. Each task starts from a published paper's own code with the paper's key contribution removed; the agent gets five hours in the repository, can request scores on development workloads, and passes only if its final patch matches or beats the paper's method on every metric of hidden held-out workloads. The paper (arXiv:2610.29932) reports 12 agents, with Claude Opus 5.5 passing 58% of its runs and 16 tasks passed by no run. More than 100 further tasks are kept private for later release, and results are mirrored on a Kaggle leaderboard.
- What it tests
- Whether agents can improve on published computer-science methods inside real research codebases, judged on hidden workloads.
- Original source
- github.com/FrontierCS/FrontierCS-2
- Last updated
- arXiv:2610.29932 (21 Oct 2026); repo last push 2026-11-17 (GitHub API, read 18 Nov 2026); Kaggle board updated 14 Nov 2026
- License
- CC-BY-4.0 (GitHub API, read 18 Nov 2026)
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed FrontierCS 2 yet. Have you run it? Share what it caught, what it missed or how you set it up.