SWE-bench vs SWE-bench-Live
SWE-bench or SWE-bench-Live? SWE-bench is the static, widely reported issue-fixing benchmark (2,294 problems from 12 Python repositories, plus a 500-problem Verified subset). SWE-bench-Live keeps adding recent GitHub issues to reduce training-data overlap and adds other languages and Windows.
| SWE-bench | SWE-bench-Live | |
|---|---|---|
| Made by | Princeton NLP & collaborators | Microsoft Research & collaborators |
| Task | Resolve a real GitHub issue so the repo's tests pass | Same task, built from issues created since 2024 |
| Size | 2,294 problems; Verified subset of 500 | 1,319 tasks at first release; Python split adds 50 verified issues a month |
| Languages | Python (multilingual and multimodal variants exist) | Python, plus MultiLang (C/C++, C#, Java, TS/JS, Go, Rust) and Windows splits |
| Contamination | Static and public, so models may have seen it | Fresh issues to lower that risk |
| Comparability | Fixed set; compare on the same split | Scores move as tasks are added; compare on the same dataset version |
| License | — | MIT |
Verdict: when to use which
Use SWE-bench if you need numbers comparable with the many published SWE-bench and SWE-bench Verified results.
Use SWE-bench-Live if you worry about memorisation or need languages beyond Python.
Sources
Other comparisons: HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp