Coding

SWT-Bench

by Niels Mündler, Mark Niklas Müller, Jingxuan He and Martin Vechev (ETH Zurich and LogicStar.ai)

What is SWT-Bench? SWT-Bench is a coding eval made by Niels Mündler, Mark Niklas Müller, Jingxuan He and Martin Vechev (ETH Zurich and LogicStar.ai). Flips the usual coding-agent task around: instead of fixing a reported GitHub bug, the model has to write a test that reproduces it. A generated test only counts if it fails on the original code and passes once the real fix is applied, and the harness also measures how much of the fixing patch the test covers. Tasks come from real Python repositories (built on SWE-bench Lite and Verified), runs happen in Docker, and there are two modes: tests merged into the project's own suite, or stand-alone reproduction scripts judged by exit code. The maintainers run submitted predictions themselves before listing them on the public leaderboard (arXiv:2406.12952).

Page last updated 2026-10-18 · Evals Review

Editor note

Flips the usual coding-agent task around: instead of fixing a reported GitHub bug, the model has to write a test that reproduces it. A generated test only counts if it fails on the original code and passes once the real fix is applied, and the harness also measures how much of the fixing patch the test covers. Tasks come from real Python repositories (built on SWE-bench Lite and Verified), runs happen in Docker, and there are two modes: tests merged into the project's own suite, or stand-alone reproduction scripts judged by exit code. The maintainers run submitted predictions themselves before listing them on the public leaderboard (arXiv:2406.12952).

What it tests
Writing issue-reproducing tests for real repositories, scored by fail-to-pass success and patch coverage.
Original source
swtbench.com/
Last updated
arXiv:2406.12952; repo logic-star-ai/swt-bench last pushed 2026-11-04 (GitHub API, read 22 Nov 2026); leaderboard entries dated up to 2026-11-03
License
MIT (code, GitHub API, read 22 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related coding evals

All Coding evals · Comparison table

Discussion

No one has discussed SWT-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions