Singularity-Bench
by Infini-AI-Lab
What is Singularity-Bench? Singularity-Bench is an agents & tool use eval made by Infini-AI-Lab. An open-ended benchmark for autonomous AI development: can an agent build the systems that make the next model? Six end-to-end tasks cover pre-training, mid-training, RL post-training, an FP8 inference engine, a coding-agent harness and multi-node serverless orchestration, each with a fixed time budget on NVIDIA GB300 GPUs and scored against a mature human-built baseline on the same hardware. Every task runs under Breakwater, the authors' hardened extension of the Harbor framework that blocks the reward-hacking attacks found in their red-teaming. The paper (arXiv:2611.16591) evaluates 11 agents from 7 models and 2 harnesses and reports Opus 5.5 with Claude Code at a Singularity-Index of 1.25, with the rest averaging 0.58. These are the authors' numbers; the tasks need large GPU budgets to reproduce.
Editor note
An open-ended benchmark for autonomous AI development: can an agent build the systems that make the next model? Six end-to-end tasks cover pre-training, mid-training, RL post-training, an FP8 inference engine, a coding-agent harness and multi-node serverless orchestration, each with a fixed time budget on NVIDIA GB300 GPUs and scored against a mature human-built baseline on the same hardware. Every task runs under Breakwater, the authors' hardened extension of the Harbor framework that blocks the reward-hacking attacks found in their red-teaming. The paper (arXiv:2611.16591) evaluates 11 agents from 7 models and 2 harnesses and reports Opus 5.5 with Claude Code at a Singularity-Index of 1.25, with the rest averaging 0.58. These are the authors' numbers; the tasks need large GPU budgets to reproduce.
- What it tests
- End-to-end AI-engineering work (training, serving, harness and cluster tasks) versus human baselines, with anti-reward-hacking safeguards.
- Original source
- infini-ai-lab.github.io/singularity-bench/
- Last updated
- arXiv:2611.16591; repo created and last pushed 2026-11-13 (GitHub API, read 16 Nov 2026)
- License
- MIT (GitHub API, read 16 Nov 2026)
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed Singularity-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.