Coding evals
SWE-bench
Gives a model a real GitHub issue from an open-source Python project and checks whether its code change makes the project's tests pass.
CodingHumanEval
A classic set of hand-written Python function problems, each checked by running hidden tests. Useful as a baseline, though top models have largely mastered it.
CodingLiveCodeBench
Keeps collecting new contest problems over time, so you can test a model only on problems released after its training cutoff.
CodingAider leaderboards (Polyglot)
Measures how well a model edits code in many programming languages inside the Aider coding assistant, including whether it follows the edit format.
CodingBigCodeBench
Programming tasks that require calling many real libraries correctly, closer to everyday developer work than puzzle-style problems.
CodingSWE-bench-Live
A continuously updated version of SWE-bench: real GitHub issues with Docker sandboxes, now covering multiple languages and operating systems, with new verified tasks added over time.