Coding

SWE-CoWork

by Xinping Lei, Minghao Liu, Ruijie Zhu et al. (TokenWave-AI)

What is SWE-CoWork? SWE-CoWork is a coding eval made by Xinping Lei, Minghao Liu, Ruijie Zhu et al. (TokenWave-AI). 101 development assignments in 101 real open-source repositories across 11 languages, where the coding agent acts like a new team member. It starts from a short handoff note, and most requirements (82% of 2,712) are held by LLM-simulated colleagues, so it has to search a project board, documents, reviews and team chat and ask the right person before implementing. Hidden fail-to-pass and pass-to-pass tests grade the result in a fresh container, with an 8-hour budget per task. The paper (arXiv:2611.20218) reports the two leading of 11 models passing about 63% of fail-to-pass tests but completing fewer than 8% of assignments.

Page last updated 2026-10-18 · Evals Review

Editor note

101 development assignments in 101 real open-source repositories across 11 languages, where the coding agent acts like a new team member. It starts from a short handoff note, and most requirements (82% of 2,712) are held by LLM-simulated colleagues, so it has to search a project board, documents, reviews and team chat and ask the right person before implementing. Hidden fail-to-pass and pass-to-pass tests grade the result in a fresh container, with an 8-hour budget per task. The paper (arXiv:2611.20218) reports the two leading of 11 models passing about 63% of fail-to-pass tests but completing fewer than 8% of assignments.

What it tests
Requirement gathering from colleagues plus end-to-end delivery of a code change in a simulated engineering team.
Original source
github.com/TokenWave-AI/swe-cowork-bench
Last updated
arXiv:2611.20218 (15 Nov 2026); repo last push 2026-11-18 (GitHub API, read 18 Nov 2026)
License
Not checked yet — see source

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related coding evals

All Coding evals · Comparison table

Discussion

No one has discussed SWE-CoWork yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions