OpenAI simple-evals
by OpenAI
What is OpenAI simple-evals? simple-evals is OpenAI's lightweight library for evaluating language models with zero-shot, chain-of-thought prompts on MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp and HealthBench. OpenAI released it to be transparent about its published accuracy numbers.
Editor note
OpenAI's small library for running MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp and HealthBench with zero-shot, chain-of-thought prompts, released to show how its published numbers were produced. Since July 2025 it gets no new models or results but keeps the reference code for HealthBench, BrowseComp and SimpleQA.
- What it tests
- Chat models on standard benchmarks with simple zero-shot prompts.
- Original source
- github.com/openai/simple-evals
- Last updated
- Repo last commit 2026-04-22
- License
- MIT
When to use OpenAI simple-evals
- You want to reproduce OpenAI-style numbers with the same simple prompts.
- You need the reference implementation of HealthBench, BrowseComp or SimpleQA.
Limitations
- Since July 2025 it is not updated for new models or results, and it does not accept new evals.
- It is not a replacement for openai/evals, which is the broader collection.
How to run it
- Clone the repository and install the API clients you need (openai, anthropic) and human-eval for HumanEval.
- List models, then run an eval with a model name and number of examples.
python -m simple-evals.simple_evals --list-models
python -m simple-evals.simple_evals --model <model_name> --examples <n>Notable results
- README (read 17 Oct 2026): deprecation notice dated July 2025; the repo keeps hosting reference implementations for HealthBench, BrowseComp and SimpleQA. [Official repository]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.