Coding

OWE-Bench (OpenWorldEngineer)

by Mercor (Jacob Cole, Cindy Yang, Aravind Selvan, Daniel Babalola and Bertie Vidgen)

What is OWE-Bench (OpenWorldEngineer)? OWE-Bench (OpenWorldEngineer) is a coding eval made by Mercor (Jacob Cole, Cindy Yang, Aravind Selvan, Daniel Babalola and Bertie Vidgen). A benchmark of 26 long-horizon engineering and research projects for AI agents, split across software engineering, physical engineering (CAD, structures, aircraft design), machine learning (GPU kernels, training, inference) and scientific computing. Some tasks start from a working codebase and ask the agent to push a measurable target such as kernel speed; others ask it to build the whole artifact from scratch. Agents get 4 to 18 hours per attempt, and each submission is scored offline by a deterministic verifier on held-out inputs with no model judge, rescaled so the untouched baseline is 0% and an expert-set ceiling is 100%. At launch Mercor reports Claude Fable 5.2 on top at 54.6%, with no model near the ceiling. Tasks are on Hugging Face and the harness is on GitHub.

Page last updated 2026-10-18 · Evals Review

Editor note

A benchmark of 26 long-horizon engineering and research projects for AI agents, split across software engineering, physical engineering (CAD, structures, aircraft design), machine learning (GPU kernels, training, inference) and scientific computing. Some tasks start from a working codebase and ask the agent to push a measurable target such as kernel speed; others ask it to build the whole artifact from scratch. Agents get 4 to 18 hours per attempt, and each submission is scored offline by a deterministic verifier on held-out inputs with no model judge, rescaled so the untouched baseline is 0% and an expert-set ceiling is 100%. At launch Mercor reports Claude Fable 5.2 on top at 54.6%, with no model near the ceiling. Tasks are on Hugging Face and the harness is on GitHub.

What it tests
Whether a coding agent can make measurable progress on multi-hour, open-ended engineering projects, with partial credit.
Original source
openworldengineer.mercor.com
Last updated
Launched 2026-11-12; HF dataset last modified 2026-11-12; harness repo last push 2026-11-12
License
Apache-2.0 (some third-party files under their own licenses)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related coding evals

All Coding evals · Comparison table

Discussion

No one has discussed OWE-Bench (OpenWorldEngineer) yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions