EvalPlus (HumanEval+ / MBPP+)
by EvalPlus team
What is EvalPlus (HumanEval+ / MBPP+)? EvalPlus is a framework that adds many more tests to HumanEval and MBPP to catch code that only passes the original, thin tests: HumanEval+ has 80x more tests than HumanEval and MBPP+ 35x more than MBPP.
Editor note
Stricter versions of HumanEval and MBPP: HumanEval+ has 80x more tests and MBPP+ 35x more, catching code that passed only the original thin tests.
- What it tests
- Function-level code generation with thorough tests.
- Original source
- github.com/evalplus/evalplus
- Last updated
- Repo last commit 2025-10-02
- License
- Apache-2.0
When to use EvalPlus (HumanEval+ / MBPP+)
- You report HumanEval or MBPP and want a stricter number.
- You want to see how much a model's score drops when weak tests are fixed.
Limitations
- Problems are short, self-contained functions, far from real repository work.
- Both base datasets are public and old, so contamination is a risk.
How to run it
- pip install evalplus.
- Generate samples, then run the evalplus evaluator on humaneval or mbpp.
pip install evalplus
evalplus.evaluate --model <model> --dataset humaneval --backend vllm --greedyNotable results
- README: HumanEval+ has 80x more tests than the original HumanEval; MBPP+ has 35x more than MBPP. [Official repository]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related coding evals
- SWE-bench — Fixing real bugs in real repositories, judged by unit tests.
- HumanEval — Writing a short correct function from a docstring.
- LiveCodeBench — Code generation, self-repair and test prediction on fresh problems.
- Aider leaderboards (Polyglot) — Practical multi-language code editing.