HELM
by Stanford CRFM
What is HELM? HELM is a rubric libraries & frameworks eval made by Stanford CRFM. Stanford's open framework and set of leaderboards for evaluating language models on many scenarios with several metrics at once, not just accuracy. Domain leaderboards include medicine (MedHELM), safety and vision-language models.
Editor note
Stanford's open framework and set of leaderboards for evaluating language models on many scenarios with several metrics at once, not just accuracy. Domain leaderboards include medicine (MedHELM), safety and vision-language models.
- What it tests
- Standardised, multi-metric comparison of models across shared scenarios.
- Original source
- crfm.stanford.edu/helm/
- Last updated
- Repo last commit 2026-06-05
- License
- Apache-2.0
When to use HELM
- You want to compare models under the same conditions on many scenarios at once, with metrics beyond accuracy such as calibration, fairness, bias, toxicity and efficiency.
- You want public leaderboards maintained by a research group: HELM Capabilities, HELM Safety and VHELM (vision-language), plus domain boards such as medicine and finance.
- You want to run the same scenarios on your own models with an open-source toolkit and inspect the raw prompts and outputs.
Limitations
- It is a broad framework, so a full run is large. Start with a few run entries and a cap on instances.
- Leaderboard results depend on the scenarios and versions chosen, so compare models within the same HELM leaderboard and release.
How to run it
- Install the crfm-helm package.
- Run one or more run entries (benchmark + model) into a suite, summarise them, then browse results in the local web server.
pip install crfm-helm
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10
helm-summarize --suite my-suite
helm-server --suite my-suiteNotable results
- The original HELM study evaluated 30 models on 42 scenarios with 7 metrics. Before HELM, models had on average been evaluated on 17.9% of its core scenarios; HELM raised this to 96.0%. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.