LiveCodeBench vs SWE-bench
LiveCodeBench or SWE-bench? LiveCodeBench tests a model on new competitive-programming problems (LeetCode, AtCoder, CodeForces) and lets you score only problems released after its training cutoff. SWE-bench tests an agent fixing real GitHub issues in Python repositories. Use LiveCodeBench for algorithmic coding with less contamination; use SWE-bench (or SWE-Bench Pro) for repository-level software engineering.
| LiveCodeBench | SWE-bench | |
|---|---|---|
| Made by | LiveCodeBench team (academic) | Princeton NLP & collaborators |
| Task | Solve a contest problem from its statement | Patch a real repository so hidden tests pass |
| Source of tasks | LeetCode, AtCoder, CodeForces, added continuously | GitHub issues from 12 Python repos (2,294; Verified: 500) |
| Contamination control | Score only problems released after a date you pick | None built in; OpenAI found Verified contaminated (Feb 2026) |
| Scenarios | Code generation, self-repair, code execution, test output prediction | Issue resolution only |
| What it needs | A model | An agent scaffold with repository access |
| Report with it | Release version and date window | Split and scaffold |
| License | — | — |
Verdict: when to use which
Use LiveCodeBench if you test a model's algorithmic coding and want to avoid problems it may have seen in training.
Use SWE-bench if you test a coding agent on real codebases; consider SWE-Bench Pro, since OpenAI stopped reporting Verified.
Dated numbers
- LiveCodeBench paper: 18 base and 34 instruction-tuned models evaluated, with findings on contamination and possible overfitting to older benchmarks like HumanEval. [LiveCodeBench paper (arXiv)]
- 23 Feb 2026: OpenAI stopped reporting SWE-bench Verified after finding flawed tests and training contamination. [OpenAI: Why we no longer evaluate SWE-bench Verified (23 Feb 2026)]
FAQ
Is LiveCodeBench contamination-free?
It reduces the risk: problems carry release dates, so you can score only those published after a model's cutoff. Report the window you used.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · Terminal-Bench vs SWE-bench · MMLU vs MMLU-Pro · GPQA vs Humanity's Last Exam · Ragas vs DeepEval · Terminal-Bench 2.1 vs Terminal-Bench 4.0 · ARC-AGI-2 vs ARC-AGI-3 · LLM-as-a-Judge vs Agent-as-a-Judge · DeepEval vs Promptfoo