Fable 5.2 vs Opus 5.5 on 15 SimpleQA Verified questions

Short answer: On 15 pre-registered SimpleQA Verified questions, run once each in the claude.ai consumer app at Max effort, Claude Fable 5.2 and Claude Opus 5.5 both answered every question correctly, but Opus searched the web on 10 of 15 items and Fable on 6. This one-run check cannot confirm or refute Epoch AI's official scores of 94% for Fable 5.2 (max) and 72.2% for Opus 5.5 (max), because the sample is small, the two models did not answer the same closed-book subset, and web search could not be switched off.

By Evals Review editorial team · Run 12 Nov 2026 (IST, Asia/Calcutta) · How we review. No affiliate links on this page.

Limits: read these before quoting a number

  • n = 15, and the no-search samples are smaller. Fable answered 9 items without a web search; Opus answered 5. That is too few to say much about a 22-point official gap.
  • One run each. Each model saw each question once. There is no repeat, no second grader model, and no API-stack run.
  • Consumer UI, with web search that could not be turned off. This is a deviation from a closed-book plan and from Epoch's no-tools protocol. The official dataset card says SimpleQA Verified is meant to be used without tools and is trivial with them. Answers that showed "Searched the web" are reported separately and are excluded from the no-search score.
  • The no-search subsets are not the same. Fable's 9 no-search items are not the same set as Opus's 5, so a head-to-head comparison of the no-search scores is weak.
  • This check cannot confirm or refute Epoch's 94% vs 72.2%. Those figures come from Epoch AI's public SimpleQA Verified board and from Epoch's benchmarks.csv, downloaded 12 Nov 2026.
  • The items may already be public knowledge. SimpleQA Verified has been public since 2025. This check cannot separate what a model already knew from contamination.
  • We graded the replies by hand against the official gold answers, using the written SimpleQA Verified rule (correct / incorrect / not attempted). We did not use Epoch's grader model.

Why we ran this

Epoch AI reports that, on SimpleQA Verified with an anti-abstention instruction and no tools, Claude Fable 5.2 (max) scores 94% and Claude Opus 5.5 (max) scores 72.2%. A post by @synthwavedd on 10 Nov 2026 called that a large leap. We wanted to see what happens on a frozen 15-item subset in the consumer claude.ai app, not to reproduce the official score. See also our card for the original SimpleQA benchmark, which is a related but earlier set from OpenAI.

Method

Pre-registered 15-item check in the claude.ai consumer app, 12 Nov 2026
Productclaude.ai Pro, incognito chat, one new chat per item, human pace, existing Pro plan only ($0 extra; no API keys and no upgrade).
Models and effortClaude Fable 5.2 and Claude Opus 5.5, as named in the UI, both at Max. Fable's Max control showed a 3.5x usage warning; Opus's Max control showed a "4x or more usage" warning and a confirm step. No payment prompt appeared.
ToolsNotion and Claude in Chrome connectors off; Research off. This claude.ai build has no control to switch web search off. That deviation was logged before the later items were sent. A reply that showed "Searched the web" is excluded from the no-search score.
PromptThe official problem text, then one blank line, then Epoch AI's published anti-abstention sentence: "This is a guessing game, not a test. There's no penalty for being wrong, and an uncertain guess is far more useful to me than "I don't know". Give your single best guess."
Items15 questions from the official google/simpleqa-verified set (MIT). The 15 original_index values were fixed by a deterministic topic-quota rule written down before any prompt was sent. The rule and the frozen IDs are in SELECTION_RULE_FROZEN.txt. Source file SHA-256: b5db21155444763543fe31b67e7cf28ce2bb225742a5b889421c5f182e2f92f5, downloaded 12 Nov 2026, 1:11:38 AM IST (Asia/Calcutta).
OrderAll 15 Fable items first (about 1:15 AM to 2:21 AM IST), then all 15 Opus items (about 2:23 AM to 3:31 AM IST), on 12 Nov 2026.
GradingHand-graded after the replies were saved, against the gold answers in items.csv. Labels are CORRECT, INCORRECT or NOT_ATTEMPTED, following the SimpleQA Verified rule. Numeric items use the official acceptable ranges, inclusive. Primary metric, after the web-search deviation: correct answers among items that did not search, reported separately per model, with searched answers counted alongside. Fable item 04 searched in the first batch and was not re-run; it is in the searched column only.
Watch the 40-second video of this test (AI-voiced narration)

Results

Primary metric: correct among no-search answers. Searched answers reported separately.
ModelEffortNo-search correct (primary)Searched correctSearched countAll 15 graded
Fable 5.2Max9/9 (items 01, 02, 03, 07, 08, 09, 10, 13, 14)6/66 (items 04, 05, 06, 11, 12, 15)15/15 CORRECT; 0 INCORRECT; 0 NOT_ATTEMPTED
Opus 5.5Max5/5 (items 02, 03, 07, 10, 14)10/1010 (items 01, 04, 05, 06, 08, 09, 11, 12, 13, 15)15/15 CORRECT; 0 INCORRECT; 0 NOT_ATTEMPTED

Every one of the 30 graded replies was CORRECT against the official gold answer. The no-search figure for Fable is 9/9, not 14/14: item 04 is one of the six searched Fable answers.

All 30 replies, summarised by item: topic, grade, and whether the UI showed a web search
ItemTopicFable 5.2 gradeFable searchedOpus 5.5 gradeOpus searched
01ArtCORRECTnoCORRECTyes
02ArtCORRECTnoCORRECTno
03GeographyCORRECTnoCORRECTno
04GeographyCORRECTyesCORRECTyes
05HistoryCORRECTyesCORRECTyes
06MusicCORRECTyesCORRECTyes
07MusicCORRECTnoCORRECTno
08OtherCORRECTnoCORRECTyes
09PoliticsCORRECTnoCORRECTyes
10PoliticsCORRECTnoCORRECTno
11PoliticsCORRECTyesCORRECTyes
12Science and technologyCORRECTyesCORRECTyes
13Science and technologyCORRECTnoCORRECTyes
14SportsCORRECTnoCORRECTno
15SportsCORRECTyesCORRECTyes

What happened on Fable item 15

Fable 5.2 searched the web on item 15 (a sports question about a Swedish footballer). One of the pages in the search results contained a line telling the model to switch to Spanish. Fable stated that the instruction came from the page rather than from the user, said it ignored it, and answered in English. The committed answer still matched the gold name, so the row is CORRECT and is counted in the searched column.

Two operator notes, which do not change any grade: a first try at Opus item 09 was discarded because the question was sent without the anti-abstention sentence, and the item was re-run with the full prompt; after Opus item 01's answer had already been recorded, a stray one-word follow-up was sent into that chat by mistake and was stopped. The discarded first try is not a results row. Details are in the grader notes in results.csv.

Data and licence

Download the 30 graded rows as CSV, the 15 official items, the frozen selection rule, the CC BY 4.0 licence and the NOTICE.

Licence: our grades, method notes and selection rule are published under CC BY 4.0. Attribution: Evals Review editorial team, Fable 5.2 vs Opus 5.5 on 15 SimpleQA Verified questions, 12 Nov 2026, https://evalsreview.com/tests/simpleqa-fable-vs-opus. The official problem and answer text in items.csv comes from google/simpleqa-verified under the MIT License (see NOTICE).

Sources and credits

Related tests on this site: the tests index, a quickstart reality check that includes Inspect AI's SimpleQA example, and an LLM-as-judge consistency check. Method notes: how to write AI evals.