NoLiMa

Editor note

A needle-in-a-haystack style long-context test where the question and the hidden fact share very few words, so a model has to make a latent association instead of matching text. The ICML 2025 paper found that 10 of 12 models tested fell below half of their short-context score by 32K tokens.

What it tests

Long-context retrieval that needs inference, not literal matching.

Original source

github.com/adobe-research/NoLiMa

Last updated

Repo last push 2025-07-17

License

Adobe Research License (non-commercial research only)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

When to use NoLiMa

• Your model or RAG setup claims a 128K+ context window and you want to know how long it stays reliable when the answer isn't a keyword match.
• Plain needle-in-a-haystack scores look perfect and you suspect literal matching is making the test too easy.
• You want an 'effective length' number: the longest context where a model keeps at least 85% of its short-context score.

Limitations

• The code and needle set use the Adobe Research License: non-commercial research only. The haystack texts have their own licences (listed in the dataset's haystack/LICENSES.md).
• Published results above 32K (64K and 128K) use fewer needle placements (11 instead of 26), so they are less precise than the shorter lengths.
• Results in the README cover models up to mid-2025 (latest update 17 Jul 2025); newer models need your own run.

How to run it

1. Install the requirements and download the needle set and haystacks from Hugging Face (amodaresi/NoLiMa) with the repo's script.
2. Serve a local model (the repo has a vLLM helper) or write a config for an API model in evaluation/model_configs; OpenRouter-style providers need tokenizer_type and tokenizer_model set.
3. Add a test config in evaluation/run_config, run ./run_tests.sh, then collect accuracy per context length with evaluation/gather_results.ipynb.
4. Other needle sets (chain-of-thought, multiple choice, direct, with distractors) live in data/needlesets.

pip install -r requirements.txt
data/download_NoLiMa_data.sh
cd evaluation/ && ./run_tests.sh

Notable results

• Paper: of 12 models that claim at least 128K context, 10 fell below 50% of their short-context baseline at 32K; GPT-4o dropped from 99.3% to 69.7%. [Paper (arXiv)]
• README results table: GPT-4.1 (claimed 1M) has an effective length of 16K; Llama 4 Scout (claimed 10M) has 1K. [Official repository]

Sources

• Paper (arXiv)
• Official repository
• Dataset (Hugging Face)

Everything above comes from the sources listed, read on 15–16 Oct 2026. Commands are copied from the official instructions; check the source for changes.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Related reasoning & knowledge evals

• GPQA — Expert-level science reasoning.
• Humanity's Last Exam — Frontier academic knowledge and reasoning.
• LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
• MMLU — World knowledge and problem solving across many school and professional subjects.

All Reasoning & knowledge evals · Comparison table