OWE-Bench (OpenWorldEngineer)
by Mercor (Jacob Cole, Cindy Yang, Aravind Selvan, Daniel Babalola and Bertie Vidgen)
What is OWE-Bench (OpenWorldEngineer)? OWE-Bench (OpenWorldEngineer) is a coding eval made by Mercor (Jacob Cole, Cindy Yang, Aravind Selvan, Daniel Babalola and Bertie Vidgen). A benchmark of 26 long-horizon engineering and research projects for AI agents, split across software engineering, physical engineering (CAD, structures, aircraft design), machine learning (GPU kernels, training, inference) and scientific computing. Some tasks start from a working codebase and ask the agent to push a measurable target such as kernel speed; others ask it to build the whole artifact from scratch. Agents get 4 to 18 hours per attempt, and each submission is scored offline by a deterministic verifier on held-out inputs with no model judge, rescaled so the untouched baseline is 0% and an expert-set ceiling is 100%. At launch Mercor reports Claude Fable 5.2 on top at 54.6%, with no model near the ceiling. Tasks are on Hugging Face and the harness is on GitHub.
Editor note
A benchmark of 26 long-horizon engineering and research projects for AI agents, split across software engineering, physical engineering (CAD, structures, aircraft design), machine learning (GPU kernels, training, inference) and scientific computing. Some tasks start from a working codebase and ask the agent to push a measurable target such as kernel speed; others ask it to build the whole artifact from scratch. Agents get 4 to 18 hours per attempt, and each submission is scored offline by a deterministic verifier on held-out inputs with no model judge, rescaled so the untouched baseline is 0% and an expert-set ceiling is 100%. At launch Mercor reports Claude Fable 5.2 on top at 54.6%, with no model near the ceiling. Tasks are on Hugging Face and the harness is on GitHub.
- What it tests
- Whether a coding agent can make measurable progress on multi-hour, open-ended engineering projects, with partial credit.
- Original source
- openworldengineer.mercor.com
- Last updated
- Launched 2026-11-12; HF dataset last modified 2026-11-12; harness repo last push 2026-11-12
- License
- Apache-2.0 (some third-party files under their own licenses)
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.
Discussion
No one has discussed OWE-Bench (OpenWorldEngineer) yet. Have you run it? Share what it caught, what it missed or how you set it up.