Coding

Infrathon

by Zhengyu Hu, Teng Xiao, Radha Poovendran et al. (University of Washington, Ai2 and collaborators)

What is Infrathon? Infrathon is a coding eval made by Zhengyu Hu, Teng Xiao, Radha Poovendran et al. (University of Washington, Ai2 and collaborators). A benchmark of long-running ML-systems engineering jobs for coding agents: building or speeding up GPU kernels, inference engines (speculative decoding, KV-cache compression, serving) and training pipelines. The agent works in a GPU container for a 12- or 24-hour budget, and a separate judge on its own GPU checks each submission for correctness against held-out workloads before scoring its speed or quality; a run keeps its best valid score. The paper covers 25 tasks and six models over 150 runs (more than 10,000 GPU-hours) and finds that model rankings change with how long the run lasts. On the project leaderboard, Claude Opus 5.5 leads with a mean rank score of 91.2. Task data is gated on Hugging Face (request access).

Page last updated 2026-10-18 · Evals Review

Editor note

A benchmark of long-running ML-systems engineering jobs for coding agents: building or speeding up GPU kernels, inference engines (speculative decoding, KV-cache compression, serving) and training pipelines. The agent works in a GPU container for a 12- or 24-hour budget, and a separate judge on its own GPU checks each submission for correctness against held-out workloads before scoring its speed or quality; a run keeps its best valid score. The paper covers 25 tasks and six models over 150 runs (more than 10,000 GPU-hours) and finds that model rankings change with how long the run lasts. On the project leaderboard, Claude Opus 5.5 leads with a mean rank score of 91.2. Task data is gated on Hugging Face (request access).

What it tests
Whether a coding agent can keep improving real AI infrastructure code over many hours without breaking correctness.
Original source
github.com/InfrathonX/Infrathon
Last updated
Repo last push 2026-11-11; arXiv v1 2026-11-09
License
PolyForm Noncommercial 1.0.0 (code); task data gated

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related coding evals

All Coding evals · Comparison table

Discussion

No one has discussed Infrathon yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions