SWE-Bench Pro

by Scale AI

What is SWE-Bench Pro? SWE-Bench Pro is Scale AI's coding-agent benchmark of long, multi-file software tasks from 41 actively maintained repositories, built to be harder and less contaminated than SWE-bench Verified. An agent gets a codebase, an issue rewritten with explicit requirements and the expected interface, and must produce a patch that passes human-reviewed tests.

Page last updated 2026-10-16 · Evals Review

Editor note

Long, multi-file software tasks from 41 maintained repositories (Python, JavaScript, TypeScript, Go). An agent must produce a patch that passes human-reviewed tests. Public set of 731 tasks (v1) or 642 re-validated tasks (V2); commercial and held-out sets stay private.

What it tests: Coding agents implementing larger, well-specified changes in real codebases.
Original source: github.com/scaleapi/SWE-bench_Pro-os
Last updated: Repo last commit 2026-09-22
License: MIT

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators.

When to use SWE-Bench Pro

You want an issue-resolution benchmark with larger changes than SWE-bench Verified: reference fixes average 107.4 lines across 4.1 files, and every task changes at least 10 lines.
You care about contamination: the public and held-out sets use GPL and other copyleft repositories, and the commercial set uses private startup code.
You need languages beyond Python: tasks cover Python, JavaScript, TypeScript and Go.

Limitations

OpenAI's July 2026 audit estimated about 30% of the 731 public v1 tasks are broken (overly strict tests, underspecified prompts, low-coverage tests) and withdrew its recommendation to adopt the benchmark.
There are several versions. v1 has 731 public tasks; V2 (22 Sep 2026) has 642 re-validated tasks plus a HARD-51 subset; third parties also publish SWE-Bench Pro Verified. Scores across versions are not comparable.
Tasks include written requirements and an interface spec, so it measures implementing a well-specified change more than resolving ambiguity. In the paper, removing them cut GPT-5's score from 25.9% to 8.4%.
Java, C++ and Rust are underrepresented, by the authors' own account.

How to run it

Load the dataset from Hugging Face (ScaleAI/SWE-bench_Pro). The default config is V2 (642 tasks); use config v1 to reproduce older 731-task numbers.
For V2, run the Harbor task directories in the official repo; for v1, use the repo's swe_bench_pro_eval.py pipeline with Docker or Modal.

from datasets import load_dataset
v2 = load_dataset('ScaleAI/SWE-bench_Pro', split='test')        # V2, 642 tasks
v1 = load_dataset('ScaleAI/SWE-bench_Pro', 'v1', split='test')  # original 731

harbor run -p v2/tasks -e modal -n 50 -a oracle

Notable results

Sep 2025 paper: 1,865 tasks in total, split into a public set (731 tasks, 11 repos), a commercial set (276, 18 private startup repos) and a held-out set (858, 12 repos). [Paper (arXiv)]
Sep 2025 paper, SWE-Agent scaffold: best public-set score 43.6% (Claude Sonnet 4.5); best commercial-set score 17.8% (Claude Opus 4.1). [Paper (arXiv)]
8 Jul 2026: on the 731-task public split, frontier models went from 23.3% to 80.3% in eight months; OpenAI's audit flagged 27.4% (pipeline) to 34.1% (human reviewers) of tasks as broken. [OpenAI: Separating signal from noise in coding evaluations (8 Jul 2026)]
22 Sep 2026: Scale released V2, 642 validated tasks across 11 repositories in Harbor format, plus the HARD-51 subset. [Official repository]
SWE-Bench Pro Verified (third-party, Sep 2026) adds anti-leakage controls and corrects 102 flawed tasks; the authors report some models score substantially lower than on the original. [SWE-Bench Pro Verified paper (arXiv)]

Sources

Everything above comes from the sources listed, read on 15–16 Oct 2026. Commands are copied from the official instructions; check the source for changes.

FAQ

What is SWE-Bench Pro?

A benchmark from Scale AI where coding agents must resolve long, multi-file issues in real repositories. It has 1,865 tasks across public, commercial and held-out sets; only the public set is downloadable.

Is SWE-Bench Pro harder than SWE-bench Verified?

It was designed to be. The paper notes 161 of Verified's 500 problems need only one- to two-line changes, while Pro excludes edits under 10 lines. When it launched, top models scored under 45% on Pro's public set versus over 70% on Verified.

Did OpenAI stop recommending SWE-Bench Pro?

Yes. On 8 Jul 2026 OpenAI estimated ~30% of the public tasks are broken and retracted its earlier recommendation, while advising developers to examine results carefully.

What is SWE-Bench Pro V2?

A re-validated public split Scale released on 22 Sep 2026: 642 tasks across 11 repositories, in Harbor format, with a HARD-51 subset. It is now the default Hugging Face config.

Which SWE-Bench Pro version should I report?

Name it exactly: v1 public (731), V2 (642), HARD-51, commercial, or the third-party Pro Verified. Their scores are not comparable.

Compare SWE-Bench Pro

SWE-Bench Pro vs SWE-bench Verified

Reviews

No reviews yet.

Reviews from practitioners who sign in with GitHub open soon.

Related coding evals

SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
HumanEval — Writing a short correct function from a docstring.
LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
Aider leaderboards (Polyglot) — Practical multi-language code editing.

All Coding evals · Comparison table