AA-Omniscience
by Artificial Analysis
What is AA-Omniscience? AA-Omniscience is a safety & factuality eval made by Artificial Analysis. A knowledge and hallucination benchmark that rewards a model for saying it does not know instead of guessing. Version 1.1 has 3,000 questions across six domains (science and maths, software engineering, finance, health, humanities and social sciences, law); a 300-question public subset is on Hugging Face and the rest stays private. The Omniscience Index runs from -100 to 100: correct answers add, wrong answers subtract, and abstentions score zero. Version 1.1 cut the set from 6,000 questions, removed saturated ones and fixed ambiguous ones, so its scores are not comparable with version 1.0.
Editor note
A knowledge and hallucination benchmark that rewards a model for saying it does not know instead of guessing. Version 1.1 has 3,000 questions across six domains (science and maths, software engineering, finance, health, humanities and social sciences, law); a 300-question public subset is on Hugging Face and the rest stays private. The Omniscience Index runs from -100 to 100: correct answers add, wrong answers subtract, and abstentions score zero. Version 1.1 cut the set from 6,000 questions, removed saturated ones and fixed ambiguous ones, so its scores are not comparable with version 1.0.
- What it tests
- Factual recall across professional domains, and whether a model abstains rather than hallucinates when unsure.
- Original source
- artificialanalysis.ai/evaluations/omniscience
- Last updated
- v1.1 public set updated 2026-10-28
- License
- Apache-2.0 (public subset)
Related safety & factuality evals
- HarmBench — Robustness to jailbreaks and harmful requests.
- TruthfulQA — Truthfulness versus imitating human misconceptions.
- StrongREJECT — Real effectiveness of jailbreak attacks.
- SimpleQA — Short-form factuality and calibration.
Discussion
No one has discussed AA-Omniscience yet. Have you run it? Share what it caught, what it missed or how you set it up.