SWE-Lancer
by OpenAI
What is SWE-Lancer? SWE-Lancer is OpenAI's benchmark of over 1,400 real freelance software tasks from Upwork, worth $1 million in total payouts. It has independent tasks, from $50 bug fixes to $32,000 features graded by end-to-end tests, and managerial tasks where the model picks the best proposal.
Editor note
Over 1,400 real Upwork freelance software tasks worth $1 million in payouts, from $50 bug fixes to $32,000 features, graded by end-to-end tests, plus managerial proposal-choice tasks.
- What it tests
- Coding agents on paid, real-world freelance work.
- Original source
- arxiv.org/abs/2502.12115
- Last updated
- Code moved to openai/preparedness
- License
- Not checked yet — see source
When to use SWE-Lancer
- You want coding-agent results expressed as money earned, not only a pass rate.
- You want end-to-end tests rather than unit tests.
Limitations
- The code moved into OpenAI's preparedness repository; check you run the current version.
- Tasks come from one company's codebase on Upwork.
How to run it
- Use the SWE-Lancer code in github.com/openai/preparedness (the original repository now points there).
Notable results
- Feb 2025 paper: over 1,400 freelance tasks from Upwork valued at $1 million in real payouts, graded with end-to-end tests triple-verified by engineers. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.