Terminal-Bench vs SWE-bench
Terminal-Bench or SWE-bench? They test different jobs. SWE-bench asks an agent to patch a real Python repository so hidden tests pass. Terminal-Bench puts the agent in a sandboxed terminal to finish end-to-end work such as building, configuring or training something, then checks the result with tests. Use both if your agent does both.
| Terminal-Bench | SWE-bench | |
|---|---|---|
| Made by | Terminal-Bench team | Princeton NLP & collaborators |
| Task | Write a patch that fixes a GitHub issue | Complete a task in a live terminal: compile, configure, train, recover |
| Environment | Docker image of the repo before the fix | Its own sandboxed container per task |
| Size | 2,294 issues; Verified subset of 500 | 2.0: 89 curated tasks; 4.0: 66 new tasks |
| Grading | Fail-to-pass and pass-to-pass tests | Hidden tests run after the agent finishes |
| Domains | Python repository bug fixing | Software, plus (in 4.0) science, ML, operations, hardware, security |
| Versions to name | Full, Lite, Verified, Multimodal, Multilingual | 2.0, 2.1, 4.0 (scores not comparable across versions) |
| License | — | — |
Verdict: when to use which
Use Terminal-Bench if your agent operates machines and shells: setup, builds, ops and long multi-step work. Name the version (2.0, 2.1 or 4.0) and harness.
Use SWE-bench if your agent edits an existing codebase to fix bugs and you want numbers comparable with many published results (keep the Verified caveats in mind).
Dated numbers
- Terminal-Bench 2.0 paper: frontier models and agents score below 65%. [Terminal-Bench 2.0 paper (arXiv)]
- Terminal-Bench 4.0 (Vals AI write-up): 66 new tasks with no overlap with 2.1's 89; each task gets 8 hours; scores are not comparable across versions. [Vals AI: Terminal-Bench 4.0]
- 23 Feb 2026: OpenAI stopped reporting SWE-bench Verified, citing flawed tests and contamination. [OpenAI: Why we no longer evaluate SWE-bench Verified (23 Feb 2026)]
FAQ
Is Terminal-Bench harder than SWE-bench?
They measure different skills, so the scores don't line up. Terminal-Bench tasks are long and end to end; SWE-bench tasks are bounded patches checked by a fixed test suite.
Which Terminal-Bench version should I cite?
Name it. 2.0 and 2.1 have 89 tasks and 4.0 has 66 new ones; scores across versions are not comparable.
Sources
- Paper (arXiv)
- Vals AI: Terminal-Bench 4.0
- OpenAI: Why we no longer evaluate SWE-bench Verified (23 Feb 2026)
- Official repository
- Harbor (runs Terminal-Bench 2.0)
- Project site
- Paper (arXiv)
- Official repository
- SWE-bench website
- OpenAI: Introducing SWE-bench Verified (2024)
- OpenAI: Separating signal from noise in coding evaluations (8 Jul 2026)
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · MMLU vs MMLU-Pro · GPQA vs Humanity's Last Exam · Ragas vs DeepEval