Coding

ExpertCrowd-SWE (EC-SWE)

by Toptal ExpertCrowd, with HUD

What is ExpertCrowd-SWE (EC-SWE)? ExpertCrowd-SWE (EC-SWE) is a coding eval made by Toptal ExpertCrowd, with HUD. 100 multi-file change requests written by practising engineers against pinned commits of 87 open-source projects in 11 languages, aimed at coding agents working in large codebases they have not seen. Each task has a sealed reference fix and its own weighted grader, mostly executable, that gives partial credit and blocks changes a maintainer would refuse to merge; every task is calibrated so the reference scores 1.0 and an untouched or inert change scores 0. The paper runs seven frontier models five times each and reports Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) on top with no statistically significant gap. 24 tasks with environments and graders are public; 76 are held out to limit contamination. Runs use the HUD platform.

Page last updated 2026-10-18 · Evals Review

Editor note

100 multi-file change requests written by practising engineers against pinned commits of 87 open-source projects in 11 languages, aimed at coding agents working in large codebases they have not seen. Each task has a sealed reference fix and its own weighted grader, mostly executable, that gives partial credit and blocks changes a maintainer would refuse to merge; every task is calibrated so the reference scores 1.0 and an untouched or inert change scores 0. The paper runs seven frontier models five times each and reports Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) on top with no statistically significant gap. 24 tasks with environments and graders are public; 76 are held out to limit contamination. Runs use the HUD platform.

What it tests
Merge-quality, multi-file changes by coding agents in unfamiliar real repositories.
Original source
huggingface.co/datasets/ExpertCrowd/ec-swe
Last updated
arXiv 2611.13246 v1 2026-11-09; HF dataset last modified 2026-11-04
License
Not checked yet — see source

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related coding evals

All Coding evals · Comparison table

Discussion

No one has discussed ExpertCrowd-SWE (EC-SWE) yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions