HumanEval vs LiveCodeBench
HumanEval or LiveCodeBench? HumanEval is a fixed, public set of short Python functions from 2021. LiveCodeBench keeps adding new LeetCode, AtCoder and CodeForces problems so you can score only problems released after a model's training cutoff.
| HumanEval | LiveCodeBench | |
|---|---|---|
| Made by | OpenAI | LiveCodeBench team (academic) |
| Task | Short function from a docstring | Contest problems, plus self-repair, code execution and test output prediction |
| Freshness | Static since 2021 | New problems added over time; pick a date window |
| Contamination | Problems are public, so models may have seen them | The paper reports contamination findings, including possible overfitting to HumanEval |
| Report with results | pass@k | Release version and date window |
| License | MIT | — |
Verdict: when to use which
Use HumanEval if you need a cheap, widely cited baseline.
Use LiveCodeBench if you want contamination-limited coding results or more than code generation.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp