Terminal-Bench 2.1 vs Terminal-Bench 4.0
Terminal-Bench 2.1 or Terminal-Bench 4.0? Report Terminal-Bench 4.0 for new results; it is the current version. 2.1 (May 2026) is the 89-task 2.0 set with 26–28 tasks fixed. 4.0 recalibrated time, CPU and memory, set an 8-hour timeout, removed 8 tasks and fixed 19. The two are not comparable, so compare scores only within one version.
| Terminal-Bench 2.1 | Terminal-Bench 4.0 | |
|---|---|---|
| Made by | Terminal-Bench team | Terminal-Bench team |
| Released | 6 May 2026 | After 3.0; current version |
| Task set | Same 89 tasks as 2.0 | Changed: 8 tasks removed (2 saturated, 2 refusals, 2 public solutions, 2 quality/platform issues) |
| Fixes | 28 tasks per the announcement (26 per the repo README): dependency drift, resource budgets, instruction-test mismatches | 19 tasks: instructions, environments and verifiers |
| Resources | Raised budgets for 8 under-resourced tasks | All tasks recalibrated; flat 8-hour agent timeout |
| Why the version jump | Point release of 2.0 | Semantic versioning: resource and task-set changes are breaking |
| Run with | Harbor (terminal-bench-2-1 dataset) | harbor run -d terminal-bench/terminal-bench@4.0.0 |
| License | — | — |
Verdict: when to use which
Use Terminal-Bench if you need to compare against results published between May 2026 and the 3.0/4.0 releases.
Use Terminal-Bench 4.0 if you are reporting a new number. State "Terminal-Bench 4.0" exactly.
Dated numbers
- 6 May 2026: 2.1 fixed 28 of 89 tasks; 9 had external dependency drift and 8 had too-small resource budgets. After the fixes, no task is unsolved. [Terminal-Bench 2.1 announcement (6 May 2026)]
- Terminal-Bench 4.0: 8 tasks removed, 19 fixed, flat 8-hour timeout; fewer timeouts and errors than 3.0. [Terminal-Bench 4.0 announcement]
- 16 Oct 2026: Claude Opus 5.5 leads the official Terminal-Bench 4.0 leaderboard at 64.85% (Claude Code, max reasoning, 330 trials); other leaderboards list it between 59.6% and 65.15% depending on harness. [Terminal-Bench leaderboard (official)]
FAQ
Is Terminal-Bench 4.0 a new benchmark?
No. The maintainers call it a continuous benchmark; 4.0 is a breaking update to the task set and resources, not a sequel.
What comes after 4.0?
4.1 (verifier improvements such as tamper-resistant verifiers) and 5.0 (new tasks), per the 4.0 announcement.
Sources
- Terminal-Bench 2.1 announcement (6 May 2026)
- Terminal-Bench 4.0 announcement
- Terminal-Bench leaderboard (official)
- Paper (arXiv)
- Official repository
- Harbor (runs Terminal-Bench 2.0)
- Project site
- Terminal-Bench 2.1 repository
- Vals AI: Terminal-Bench 4.0
- Snorkel AI: Terminal-Bench 4.0
- Artificial Analysis: Terminal-Bench 4.0
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · Terminal-Bench vs SWE-bench · MMLU vs MMLU-Pro · GPQA vs Humanity's Last Exam · Ragas vs DeepEval · ARC-AGI-2 vs ARC-AGI-3 · LiveCodeBench vs SWE-bench · LLM-as-a-Judge vs Agent-as-a-Judge · DeepEval vs Promptfoo