SWE-CoWork
by Xinping Lei, Minghao Liu, Ruijie Zhu et al. (TokenWave-AI)
What is SWE-CoWork? SWE-CoWork is a coding eval made by Xinping Lei, Minghao Liu, Ruijie Zhu et al. (TokenWave-AI). 101 development assignments in 101 real open-source repositories across 11 languages, where the coding agent acts like a new team member. It starts from a short handoff note, and most requirements (82% of 2,712) are held by LLM-simulated colleagues, so it has to search a project board, documents, reviews and team chat and ask the right person before implementing. Hidden fail-to-pass and pass-to-pass tests grade the result in a fresh container, with an 8-hour budget per task. The paper (arXiv:2611.20218) reports the two leading of 11 models passing about 63% of fail-to-pass tests but completing fewer than 8% of assignments.
Editor note
101 development assignments in 101 real open-source repositories across 11 languages, where the coding agent acts like a new team member. It starts from a short handoff note, and most requirements (82% of 2,712) are held by LLM-simulated colleagues, so it has to search a project board, documents, reviews and team chat and ask the right person before implementing. Hidden fail-to-pass and pass-to-pass tests grade the result in a fresh container, with an 8-hour budget per task. The paper (arXiv:2611.20218) reports the two leading of 11 models passing about 63% of fail-to-pass tests but completing fewer than 8% of assignments.
- What it tests
- Requirement gathering from colleagues plus end-to-end delivery of a code change in a simulated engineering team.
- Original source
- github.com/TokenWave-AI/swe-cowork-bench
- Last updated
- arXiv:2611.20218 (15 Nov 2026); repo last push 2026-11-18 (GitHub API, read 18 Nov 2026)
- License
- Not checked yet — see source
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed SWE-CoWork yet. Have you run it? Share what it caught, what it missed or how you set it up.