SWT-Bench
by Niels Mündler, Mark Niklas Müller, Jingxuan He and Martin Vechev (ETH Zurich and LogicStar.ai)
What is SWT-Bench? SWT-Bench is a coding eval made by Niels Mündler, Mark Niklas Müller, Jingxuan He and Martin Vechev (ETH Zurich and LogicStar.ai). Flips the usual coding-agent task around: instead of fixing a reported GitHub bug, the model has to write a test that reproduces it. A generated test only counts if it fails on the original code and passes once the real fix is applied, and the harness also measures how much of the fixing patch the test covers. Tasks come from real Python repositories (built on SWE-bench Lite and Verified), runs happen in Docker, and there are two modes: tests merged into the project's own suite, or stand-alone reproduction scripts judged by exit code. The maintainers run submitted predictions themselves before listing them on the public leaderboard (arXiv:2406.12952).
Editor note
Flips the usual coding-agent task around: instead of fixing a reported GitHub bug, the model has to write a test that reproduces it. A generated test only counts if it fails on the original code and passes once the real fix is applied, and the harness also measures how much of the fixing patch the test covers. Tasks come from real Python repositories (built on SWE-bench Lite and Verified), runs happen in Docker, and there are two modes: tests merged into the project's own suite, or stand-alone reproduction scripts judged by exit code. The maintainers run submitted predictions themselves before listing them on the public leaderboard (arXiv:2406.12952).
- What it tests
- Writing issue-reproducing tests for real repositories, scored by fail-to-pass success and patch coverage.
- Original source
- swtbench.com/
- Last updated
- arXiv:2406.12952; repo logic-star-ai/swt-bench last pushed 2026-11-04 (GitHub API, read 22 Nov 2026); leaderboard entries dated up to 2026-11-03
- License
- MIT (code, GitHub API, read 22 Nov 2026)
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed SWT-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.