Coding benchmarks for LLMs, compared
Coding benchmarks range from short function-writing tasks to fixing real issues in real repositories. Use the table to match a benchmark to the kind of code your model has to write.
| Eval | Made by | What it tests | License | Last updated | Source |
|---|---|---|---|---|---|
| SWE-bench | Princeton NLP & collaborators | Fixing real bugs in real repositories, judged by unit tests. | — | — | swebench.com |
| HumanEval | OpenAI | Writing a short correct function from a docstring. | MIT | Repo last commit 2025-01-17 | github.com |
| LiveCodeBench | LiveCodeBench team (academic) | Code generation, self-repair and test prediction on fresh problems. | — | — | livecodebench.github.io |
| Aider leaderboards (Polyglot) | Aider | Practical multi-language code editing. | — | — | aider.chat |
| BigCodeBench | BigCode project | Library-heavy, practical Python tasks. | Apache-2.0 | Repo last commit 2026-01-03 | github.com |
| SWE-bench-Live | Microsoft Research & collaborators | Fixing live software issues across languages and platforms, judged by tests in Docker. | MIT | Repo last commit 2026-09-25 | swe-bench-live.github.io |
In short
SWE-bench — Gives a model a real GitHub issue from an open-source Python project and checks whether its code change makes the project's tests pass.
HumanEval — A classic set of hand-written Python function problems, each checked by running hidden tests. Useful as a baseline, though top models have largely mastered it.
LiveCodeBench — Keeps collecting new contest problems over time, so you can test a model only on problems released after its training cutoff.
Aider leaderboards (Polyglot) — Measures how well a model edits code in many programming languages inside the Aider coding assistant, including whether it follows the edit format.
BigCodeBench — Programming tasks that require calling many real libraries correctly, closer to everyday developer work than puzzle-style problems.
SWE-bench-Live — A continuously updated version of SWE-bench: real GitHub issues with Docker sandboxes, now covering multiple languages and operating systems, with new verified tasks added over time.
Browse the full Coding category, or see other comparisons.