HELMET
by Princeton NLP
What is HELMET? HELMET is a reasoning & knowledge eval made by Princeton NLP. A long-context benchmark with seven application-style categories, controllable input lengths up to 128K tokens, model-based scoring and few-shot prompts so base models can be tested too. Its study of 59 models found that needle-in-a-haystack scores do not reliably predict real long-context performance.
Editor note
A long-context benchmark with seven application-style categories, controllable input lengths up to 128K tokens, model-based scoring and few-shot prompts so base models can be tested too. Its study of 59 models found that needle-in-a-haystack scores do not reliably predict real long-context performance.
- What it tests
- Long-context models on recall, RAG, re-ranking, citation, long-document QA, summarisation and in-context learning.
- Original source
- github.com/princeton-nlp/HELMET
- Last updated
- Repo last commit 2026-09-19
- License
- MIT
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related reasoning & knowledge evals
- GPQA — Expert-level science reasoning.
- Humanity's Last Exam — Frontier academic knowledge and reasoning.
- LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge).
- MMLU — World knowledge and problem solving across many school and professional subjects.