lm-evaluation-harness
by EleutherAI
What is lm-evaluation-harness? lm-evaluation-harness is EleutherAI's open-source (MIT) framework for running standard academic benchmarks against language models, with over 60 benchmarks and hundreds of subtasks and variants. It powered Hugging Face's Open LLM Leaderboard.
Editor note
An open-source framework for running over 60 standard academic benchmarks, with hundreds of subtasks and variants, against language models. It powered Hugging Face's Open LLM Leaderboard.
- What it tests
- Models on standard academic benchmarks with consistent prompts.
- Original source
- github.com/EleutherAI/lm-evaluation-harness
- Last updated
- Repo last commit 2026-09-14
- License
- MIT
When to use lm-evaluation-harness
- You want to run many standard benchmarks (MMLU, GSM8K, HellaSwag and others) on a model with one tool and consistent prompts.
- You compare base or open-weight models and want results others can reproduce.
Limitations
- Scores depend on task version, prompt format and few-shot settings; report them with the harness version.
- It targets model benchmarks, not tests of your own app; use DeepEval or Promptfoo for that.
How to run it
- Install from the repository.
- Run lm_eval with a model and a task list.
pip install lm-eval
lm_eval --model hf --model_args pretrained=<model> --tasks gsm8k,mmlu --batch_size 8Notable results
- README (read 16 Oct 2026): over 60 standard academic benchmarks with hundreds of subtasks; the backend for Hugging Face's Open LLM Leaderboard and used in hundreds of papers. [Official repository]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.