Infrathon
by Zhengyu Hu, Teng Xiao, Radha Poovendran et al. (University of Washington, Ai2 and collaborators)
What is Infrathon? Infrathon is a coding eval made by Zhengyu Hu, Teng Xiao, Radha Poovendran et al. (University of Washington, Ai2 and collaborators). A benchmark of long-running ML-systems engineering jobs for coding agents: building or speeding up GPU kernels, inference engines (speculative decoding, KV-cache compression, serving) and training pipelines. The agent works in a GPU container for a 12- or 24-hour budget, and a separate judge on its own GPU checks each submission for correctness against held-out workloads before scoring its speed or quality; a run keeps its best valid score. The paper covers 25 tasks and six models over 150 runs (more than 10,000 GPU-hours) and finds that model rankings change with how long the run lasts. On the project leaderboard, Claude Opus 5.5 leads with a mean rank score of 91.2. Task data is gated on Hugging Face (request access).
Editor note
A benchmark of long-running ML-systems engineering jobs for coding agents: building or speeding up GPU kernels, inference engines (speculative decoding, KV-cache compression, serving) and training pipelines. The agent works in a GPU container for a 12- or 24-hour budget, and a separate judge on its own GPU checks each submission for correctness against held-out workloads before scoring its speed or quality; a run keeps its best valid score. The paper covers 25 tasks and six models over 150 runs (more than 10,000 GPU-hours) and finds that model rankings change with how long the run lasts. On the project leaderboard, Claude Opus 5.5 leads with a mean rank score of 91.2. Task data is gated on Hugging Face (request access).
- What it tests
- Whether a coding agent can keep improving real AI infrastructure code over many hours without breaking correctness.
- Original source
- github.com/InfrathonX/Infrathon
- Last updated
- Repo last push 2026-11-11; arXiv v1 2026-11-09
- License
- PolyForm Noncommercial 1.0.0 (code); task data gated
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed Infrathon yet. Have you run it? Share what it caught, what it missed or how you set it up.