SWE-Bench Pro vs SWE-bench Verified

SWE-Bench Pro or SWE-bench Verified? Neither gives a clean headline number in October 2026. SWE-bench Verified (500 Python issues) is saturated and contaminated, and OpenAI stopped reporting it in February 2026. SWE-Bench Pro (731 public tasks in v1, 642 in V2) is harder and covers four languages, but OpenAI estimated ~30% of its v1 public tasks are broken. If you report one, report SWE-Bench Pro V2 and name the version.

Last updated 2026-10-16 · Evals Review

	SWE-Bench Pro	SWE-bench Verified
Made by	Scale AI	Princeton NLP & collaborators
Made of	Long, multi-file changes from 41 repos (public set from 11 GPL/copyleft repos)	Issues from 12 open-source Python repos, filtered by engineers
Size	1,865 total; public 731 (v1) or 642 (V2); 276 commercial and 858 held-out kept private	500 problems (from 1,699 reviewed, 3 experts each)
Change size	At least 10 lines; reference fixes average 107.4 lines over 4.1 files	161 of 500 need only 1–2 line changes (per the Pro paper)
Languages	Python, JavaScript, TypeScript, Go	Python
Task prompt	Issue plus written requirements and interface spec	Original issue text only
Contamination	Copyleft and private code to limit training overlap; OpenAI found some cases, rarer	Frontier models reproduced gold patches from memory (OpenAI, Feb 2026)
Known test problems	~30% of v1 public tasks broken (OpenAI, Jul 2026); V2 re-validated	Flawed tests in 59.4% of 138 audited hard problems
Status, Oct 2026	V2 is the default dataset config since 22 Sep 2026	OpenAI no longer reports it
License	MIT	—

Verdict: when to use which

Use SWE-Bench Pro if you need a coding-agent number that still separates frontier models. Use V2 or HARD-51, state the version and scaffold, and don't mix with v1 numbers.

Use SWE-bench Verified if you are comparing against older published results or smaller models whose scores are far from the ceiling. Treat differences at the top as noise.

Dated numbers

23 Feb 2026: the best SWE-bench Verified score rose only from 74.9% to 80.9% in the previous six months; OpenAI stopped reporting it. [OpenAI: Why we no longer evaluate SWE-bench Verified (23 Feb 2026)]
Sep 2025: at launch, the best public-set SWE-Bench Pro score was 43.6% (Claude Sonnet 4.5, SWE-Agent); the best commercial-set score was 17.8%. [SWE-Bench Pro paper (arXiv)]
8 Jul 2026: on Pro's 731-task public split, frontier models went from 23.3% to 80.3% in eight months; 27.4% (agent pipeline) to 34.1% (human reviewers) of tasks were judged broken. [OpenAI: Separating signal from noise in coding evaluations (8 Jul 2026)]
22 Sep 2026: Scale released SWE-Bench Pro V2 with 642 validated tasks and a HARD-51 subset. [SWE-Bench Pro repository]

FAQ

Why did OpenAI stop using SWE-bench Verified?

Its February 2026 audit found flawed tests in 59.4% of 138 hard problems and evidence that frontier models had seen some problems and fixes during training.

Is SWE-Bench Pro contamination-free?

No. Pro uses copyleft and private code to limit overlap; OpenAI found contamination there too, but rarer, and no model reproduced a complete gold patch.

Are SWE-Bench Pro v1 and V2 scores comparable?

No. V2 has 642 re-validated tasks versus 731 in v1, so compare only within one version.

Sources

Every row restates facts from these sources, also cited on each eval's page. A dash means not verified.

Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · Terminal-Bench vs SWE-bench · MMLU vs MMLU-Pro · GPQA vs Humanity's Last Exam · Ragas vs DeepEval