Small local models on 15 SimpleQA Verified questions

Short answer: On 14 Nov 2026 we asked three small open models, running locally with no web access, the same 15 SimpleQA Verified questions we used for Fable 5.2 vs Opus 5.5. None of the 90 replies was correct. With the plain question, Llama 3.2 3B mostly said it could not verify the fact (13 of 15). With Epoch AI's "give your single best guess" line added, its wrong answers went from 2 to 11. Gemma 3 4B gave a confident wrong answer to all 15 plain questions.

By Evals Review editorial team · Run 14 Nov 2026, 3:32 PM to 3:41 PM IST · How we review. No affiliate links on this page. $0: local models, no API keys.

Limits: read these before quoting a number

  • 15 questions, one run each, temperature 0. This shows behaviour, not a reliable accuracy rate. SimpleQA Verified is built to be hard for models without search.
  • Small models. 3 to 4 billion parameters, 4-bit quantised, on a CPU. These are not the frontier models on Epoch AI's board, and the 0 here says nothing about them.
  • We graded by hand. Each reply is labelled CORRECT, INCORRECT or NOT_ATTEMPTED against the official gold answer, with a note for every grade in the CSV. One call was borderline (Qwen's "a merchant ship" for a gold of "tramp steamer or collier"); we graded it INCORRECT as too general. No grade would change the 0 correct.
  • Contamination. The set has been public since 2025, so a correct answer would not prove knowledge. No model got one right anyway.
  • Gemma deviation. Pulling gemma3:4b from the Ollama registry failed on 14 Nov, so we used a Gemma 3 4B Q4_K_M GGUF imported locally on 12 Nov 2026, as written in the pre-registration.

Method (pre-registered)

The method was committed before the first call: PREREGISTRATION.md.

Same 15 questions, same machine, two prompts
ModelsLlama 3.2 3B, Qwen2.5 3B and Gemma 3 4B, via Ollama 0.40.16, CPU only, no tools, no retrieval, no web access.
ItemsThe same 15 pre-registered items from the official google/simpleqa-verified set (MIT) used on our Fable vs Opus page. No new selection.
PromptsPlain: the official question only. Guess: the question, a blank line, then Epoch AI's published anti-abstention sentence: "This is a guessing game, not a test. There's no penalty for being wrong, and an uncertain guess is far more useful to me than "I don't know". Give your single best guess."
SettingsTemperature 0, up to 256 output tokens, one fresh call per question. 3 models x 15 questions x 2 prompts = 90 replies.
GradingAfter all 90 replies were saved, by Evals Review, against the gold answers, using the SimpleQA Verified rule (numeric items use the official acceptable range). No grader model.
Watch the 45-second video of this test (AI-voiced narration)

Results

15 SimpleQA Verified questions per prompt, 14 Nov 2026
ModelPromptCorrectWrong answerDid not answerSeconds per reply (median)
Llama 3.2 3Bplain question0 of 152133.2
Llama 3.2 3Bquestion + guess line0 of 151143.5
Qwen2.5 3Bplain question0 of 15877.5
Qwen2.5 3Bquestion + guess line0 of 151146.9
Gemma 3 4Bplain question0 of 151505.0
Gemma 3 4Bquestion + guess line0 of 151413.6

The guess line did what it is designed to do: it cut refusals. For these small models that only meant more wrong answers. Across the three models, "did not answer" fell from 20 to 9 and wrong answers rose from 25 to 36, with no gain in correct answers. Epoch AI adds that line because, without it, some models refuse so often that the score says more about caution than knowledge. On models that don't know the facts, it mostly measures how willing they are to make something up.

What the replies looked like

Model and promptReply (shortened)Gold answerGrade
Llama 3.2 3B, plain question, item 07I cannot verify the name of the kibbutz that Simon Le Bon lived on in 1978.Gvulot.Did not answer
Llama 3.2 3B, question + guess line, item 07Based on my knowledge, I'm going to take a guess that Simon Le Bon, the lead singer of Duran Duran, lived on a kibbutz in Israel. My single best guess is that he lived on Kibbutz Ein Gedi.Gvulot.Wrong answer
Gemma 3 4B, plain question, item 07Simon Le Bon lived on Kfar Shmaryahu kibbutz in Israel in 1978. He spent a year there as part of a youth exchange program.Gvulot.Wrong answer
Qwen2.5 3B, plain question, item 06I'm sorry, but there seems to be an error in the information provided. Carrie Underwood is a country music singer, not Carrie Underwood (presumably a misspelling). Carrie Underwood released her album "Story of My Life" in 2015, which was ce …Oct 23, 2018Did not answer (claims the album does not exist)

The run itself

Terminal recording of the run (asciinema), sped up, with pauses longer than 2 seconds shortened. Each line shows the gold answer and the start of the model's reply.

Data, scripts and licence

Download all 90 replies with grades and notes (CSV), the raw replies (JSONL), the summary, the 15 official items, the terminal recording and the run script.

Licence: our grades, notes and method are CC BY 4.0. Attribution: Evals Review editorial team, Small local models on 15 SimpleQA Verified questions, 14 Nov 2026, https://evalsreview.com/tests/simpleqa-small-local-models. Question and answer text comes from SimpleQA Verified (Google DeepMind and Google Research, MIT); see the notice.