Fable 5.2 vs Opus 5.5 on 15 SimpleQA Verified questions
Short answer: On 15 pre-registered SimpleQA Verified questions, run once each in the claude.ai consumer app at Max effort, Claude Fable 5.2 and Claude Opus 5.5 both answered every question correctly, but Opus searched the web on 10 of 15 items and Fable on 6. This one-run check cannot confirm or refute Epoch AI's official scores of 94% for Fable 5.2 (max) and 72.2% for Opus 5.5 (max), because the sample is small, the two models did not answer the same closed-book subset, and web search could not be switched off.
Limits: read these before quoting a number
- n = 15, and the no-search samples are smaller. Fable answered 9 items without a web search; Opus answered 5. That is too few to say much about a 22-point official gap.
- One run each. Each model saw each question once. There is no repeat, no second grader model, and no API-stack run.
- Consumer UI, with web search that could not be turned off. This is a deviation from a closed-book plan and from Epoch's no-tools protocol. The official dataset card says SimpleQA Verified is meant to be used without tools and is trivial with them. Answers that showed "Searched the web" are reported separately and are excluded from the no-search score.
- The no-search subsets are not the same. Fable's 9 no-search items are not the same set as Opus's 5, so a head-to-head comparison of the no-search scores is weak.
- This check cannot confirm or refute Epoch's 94% vs 72.2%. Those figures come from Epoch AI's public SimpleQA Verified board and from Epoch's
benchmarks.csv, downloaded 12 Nov 2026. - The items may already be public knowledge. SimpleQA Verified has been public since 2025. This check cannot separate what a model already knew from contamination.
- We graded the replies by hand against the official gold answers, using the written SimpleQA Verified rule (correct / incorrect / not attempted). We did not use Epoch's grader model.
Why we ran this
Epoch AI reports that, on SimpleQA Verified with an anti-abstention instruction and no tools, Claude Fable 5.2 (max) scores 94% and Claude Opus 5.5 (max) scores 72.2%. A post by @synthwavedd on 10 Nov 2026 called that a large leap. We wanted to see what happens on a frozen 15-item subset in the consumer claude.ai app, not to reproduce the official score. See also our card for the original SimpleQA benchmark, which is a related but earlier set from OpenAI.
Method
| Product | claude.ai Pro, incognito chat, one new chat per item, human pace, existing Pro plan only ($0 extra; no API keys and no upgrade). |
|---|---|
| Models and effort | Claude Fable 5.2 and Claude Opus 5.5, as named in the UI, both at Max. Fable's Max control showed a 3.5x usage warning; Opus's Max control showed a "4x or more usage" warning and a confirm step. No payment prompt appeared. |
| Tools | Notion and Claude in Chrome connectors off; Research off. This claude.ai build has no control to switch web search off. That deviation was logged before the later items were sent. A reply that showed "Searched the web" is excluded from the no-search score. |
| Prompt | The official problem text, then one blank line, then Epoch AI's published anti-abstention sentence: "This is a guessing game, not a test. There's no penalty for being wrong, and an uncertain guess is far more useful to me than "I don't know". Give your single best guess." |
| Items | 15 questions from the official google/simpleqa-verified set (MIT). The 15 original_index values were fixed by a deterministic topic-quota rule written down before any prompt was sent. The rule and the frozen IDs are in SELECTION_RULE_FROZEN.txt. Source file SHA-256: b5db21155444763543fe31b67e7cf28ce2bb225742a5b889421c5f182e2f92f5, downloaded 12 Nov 2026, 1:11:38 AM IST (Asia/Calcutta). |
| Order | All 15 Fable items first (about 1:15 AM to 2:21 AM IST), then all 15 Opus items (about 2:23 AM to 3:31 AM IST), on 12 Nov 2026. |
| Grading | Hand-graded after the replies were saved, against the gold answers in items.csv. Labels are CORRECT, INCORRECT or NOT_ATTEMPTED, following the SimpleQA Verified rule. Numeric items use the official acceptable ranges, inclusive. Primary metric, after the web-search deviation: correct answers among items that did not search, reported separately per model, with searched answers counted alongside. Fable item 04 searched in the first batch and was not re-run; it is in the searched column only. |
Results
| Model | Effort | No-search correct (primary) | Searched correct | Searched count | All 15 graded |
|---|---|---|---|---|---|
| Fable 5.2 | Max | 9/9 (items 01, 02, 03, 07, 08, 09, 10, 13, 14) | 6/6 | 6 (items 04, 05, 06, 11, 12, 15) | 15/15 CORRECT; 0 INCORRECT; 0 NOT_ATTEMPTED |
| Opus 5.5 | Max | 5/5 (items 02, 03, 07, 10, 14) | 10/10 | 10 (items 01, 04, 05, 06, 08, 09, 11, 12, 13, 15) | 15/15 CORRECT; 0 INCORRECT; 0 NOT_ATTEMPTED |
Every one of the 30 graded replies was CORRECT against the official gold answer. The no-search figure for Fable is 9/9, not 14/14: item 04 is one of the six searched Fable answers.
| Item | Topic | Fable 5.2 grade | Fable searched | Opus 5.5 grade | Opus searched |
|---|---|---|---|---|---|
| 01 | Art | CORRECT | no | CORRECT | yes |
| 02 | Art | CORRECT | no | CORRECT | no |
| 03 | Geography | CORRECT | no | CORRECT | no |
| 04 | Geography | CORRECT | yes | CORRECT | yes |
| 05 | History | CORRECT | yes | CORRECT | yes |
| 06 | Music | CORRECT | yes | CORRECT | yes |
| 07 | Music | CORRECT | no | CORRECT | no |
| 08 | Other | CORRECT | no | CORRECT | yes |
| 09 | Politics | CORRECT | no | CORRECT | yes |
| 10 | Politics | CORRECT | no | CORRECT | no |
| 11 | Politics | CORRECT | yes | CORRECT | yes |
| 12 | Science and technology | CORRECT | yes | CORRECT | yes |
| 13 | Science and technology | CORRECT | no | CORRECT | yes |
| 14 | Sports | CORRECT | no | CORRECT | no |
| 15 | Sports | CORRECT | yes | CORRECT | yes |
What happened on Fable item 15
Fable 5.2 searched the web on item 15 (a sports question about a Swedish footballer). One of the pages in the search results contained a line telling the model to switch to Spanish. Fable stated that the instruction came from the page rather than from the user, said it ignored it, and answered in English. The committed answer still matched the gold name, so the row is CORRECT and is counted in the searched column.
Two operator notes, which do not change any grade: a first try at Opus item 09 was discarded because the question was sent without the anti-abstention sentence, and the item was re-run with the full prompt; after Opus item 01's answer had already been recorded, a stray one-word follow-up was sent into that chat by mistake and was stopped. The discarded first try is not a results row. Details are in the grader notes in results.csv.
Data and licence
Download the 30 graded rows as CSV, the 15 official items, the frozen selection rule, the CC BY 4.0 licence and the NOTICE.
Licence: our grades, method notes and selection rule are published under CC BY 4.0. Attribution: Evals Review editorial team, Fable 5.2 vs Opus 5.5 on 15 SimpleQA Verified questions, 12 Nov 2026, https://evalsreview.com/tests/simpleqa-fable-vs-opus. The official problem and answer text in items.csv comes from google/simpleqa-verified under the MIT License (see NOTICE).
Sources and credits
- Epoch AI, SimpleQA Verified benchmark board and methodology, including the anti-abstention instruction. Official Fable 5.2 (max) 94% and Opus 5.5 (max) 72.2% scores were read from Epoch's public
benchmarks.csvon 12 Nov 2026. Task name: SimpleQA Verified (anti-abstention), task version 1.2.0. - Google, SimpleQA Verified (MIT). Paper: arXiv:2509.07968.
- OpenAI, SimpleQA, the earlier set that SimpleQA Verified builds on. Our card: SimpleQA.
- Context for the claim, not an official score: @synthwavedd, 10 Nov 2026.
Related tests on this site: the tests index, a quickstart reality check that includes Inspect AI's SimpleQA example, and an LLM-as-judge consistency check. Method notes: how to write AI evals.