ExpertCrowd-SWE (EC-SWE)
by Toptal ExpertCrowd, with HUD
What is ExpertCrowd-SWE (EC-SWE)? ExpertCrowd-SWE (EC-SWE) is a coding eval made by Toptal ExpertCrowd, with HUD. 100 multi-file change requests written by practising engineers against pinned commits of 87 open-source projects in 11 languages, aimed at coding agents working in large codebases they have not seen. Each task has a sealed reference fix and its own weighted grader, mostly executable, that gives partial credit and blocks changes a maintainer would refuse to merge; every task is calibrated so the reference scores 1.0 and an untouched or inert change scores 0. The paper runs seven frontier models five times each and reports Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) on top with no statistically significant gap. 24 tasks with environments and graders are public; 76 are held out to limit contamination. Runs use the HUD platform.
Editor note
100 multi-file change requests written by practising engineers against pinned commits of 87 open-source projects in 11 languages, aimed at coding agents working in large codebases they have not seen. Each task has a sealed reference fix and its own weighted grader, mostly executable, that gives partial credit and blocks changes a maintainer would refuse to merge; every task is calibrated so the reference scores 1.0 and an untouched or inert change scores 0. The paper runs seven frontier models five times each and reports Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) on top with no statistically significant gap. 24 tasks with environments and graders are public; 76 are held out to limit contamination. Runs use the HUD platform.
- What it tests
- Merge-quality, multi-file changes by coding agents in unfamiliar real repositories.
- Original source
- huggingface.co/datasets/ExpertCrowd/ec-swe
- Last updated
- arXiv 2611.13246 v1 2026-11-09; HF dataset last modified 2026-11-04
- License
- Not checked yet — see source
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed ExpertCrowd-SWE (EC-SWE) yet. Have you run it? Share what it caught, what it missed or how you set it up.