Agents & tool use

Singularity-Bench

by Infini-AI-Lab

What is Singularity-Bench? Singularity-Bench is an agents & tool use eval made by Infini-AI-Lab. An open-ended benchmark for autonomous AI development: can an agent build the systems that make the next model? Six end-to-end tasks cover pre-training, mid-training, RL post-training, an FP8 inference engine, a coding-agent harness and multi-node serverless orchestration, each with a fixed time budget on NVIDIA GB300 GPUs and scored against a mature human-built baseline on the same hardware. Every task runs under Breakwater, the authors' hardened extension of the Harbor framework that blocks the reward-hacking attacks found in their red-teaming. The paper (arXiv:2611.16591) evaluates 11 agents from 7 models and 2 harnesses and reports Opus 5.5 with Claude Code at a Singularity-Index of 1.25, with the rest averaging 0.58. These are the authors' numbers; the tasks need large GPU budgets to reproduce.

Page last updated 2026-10-18 · Evals Review

Editor note

An open-ended benchmark for autonomous AI development: can an agent build the systems that make the next model? Six end-to-end tasks cover pre-training, mid-training, RL post-training, an FP8 inference engine, a coding-agent harness and multi-node serverless orchestration, each with a fixed time budget on NVIDIA GB300 GPUs and scored against a mature human-built baseline on the same hardware. Every task runs under Breakwater, the authors' hardened extension of the Harbor framework that blocks the reward-hacking attacks found in their red-teaming. The paper (arXiv:2611.16591) evaluates 11 agents from 7 models and 2 harnesses and reports Opus 5.5 with Claude Code at a Singularity-Index of 1.25, with the rest averaging 0.58. These are the authors' numbers; the tasks need large GPU budgets to reproduce.

What it tests
End-to-end AI-engineering work (training, serving, harness and cluster tasks) versus human baselines, with anti-reward-hacking safeguards.
Original source
infini-ai-lab.github.io/singularity-bench/
Last updated
arXiv:2611.16591; repo created and last pushed 2026-11-13 (GitHub API, read 16 Nov 2026)
License
MIT (GitHub API, read 16 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related agents & tool use evals

  • GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
  • WebArena — Autonomous web navigation and task completion.
  • OSWorld — Computer-use agents operating real operating systems and apps.
  • Terminal-Bench — Agents working in the shell end to end.

All Agents & tool use evals · Comparison table

Discussion

No one has discussed Singularity-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions