FACTS Grounding
by Google DeepMind & Google Research
What is FACTS Grounding? FACTS Grounding is a benchmark and Kaggle leaderboard from Google DeepMind and Google Research that tests whether a model's long-form answer is fully grounded in a document supplied in the prompt (up to 32K tokens) while still meeting the user's request.
Editor note
Each prompt pairs a user request with a document of up to 32K tokens; the long answer must meet the request and be fully supported by the document. Several judge models score it, and the Kaggle leaderboard keeps a private split next to 860 public examples.
- What it tests
- Whether long answers stay grounded in a supplied document.
- Original source
- arxiv.org/abs/2501.03200
- Last updated
- Not checked yet
- License
- CC-BY-4.0
When to use FACTS Grounding
- You build RAG or document-QA features and care about answers that stay within the source.
- You want a leaderboard with a private split, which limits overfitting.
Limitations
- Scoring relies on judge models; the final score averages several judges to reduce bias, but it is still automated.
- Only 860 examples are public; the full leaderboard uses a private split you cannot download.
How to run it
- Load the 860 public examples from Hugging Face (google/FACTS-grounding-public).
- Use the evaluation starter code on Kaggle, linked from the dataset card, to generate answers and run the judge prompts.
Notable results
- Paper: responses are first disqualified if they do not fulfil the user request, then judged accurate only if fully grounded in the document; the score aggregates multiple judge models. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.