Small local models on 15 SimpleQA Verified questions
Short answer: On 14 Nov 2026 we asked three small open models, running locally with no web access, the same 15 SimpleQA Verified questions we used for Fable 5.2 vs Opus 5.5. None of the 90 replies was correct. With the plain question, Llama 3.2 3B mostly said it could not verify the fact (13 of 15). With Epoch AI's "give your single best guess" line added, its wrong answers went from 2 to 11. Gemma 3 4B gave a confident wrong answer to all 15 plain questions.
Limits: read these before quoting a number
- 15 questions, one run each, temperature 0. This shows behaviour, not a reliable accuracy rate. SimpleQA Verified is built to be hard for models without search.
- Small models. 3 to 4 billion parameters, 4-bit quantised, on a CPU. These are not the frontier models on Epoch AI's board, and the 0 here says nothing about them.
- We graded by hand. Each reply is labelled CORRECT, INCORRECT or NOT_ATTEMPTED against the official gold answer, with a note for every grade in the CSV. One call was borderline (Qwen's "a merchant ship" for a gold of "tramp steamer or collier"); we graded it INCORRECT as too general. No grade would change the 0 correct.
- Contamination. The set has been public since 2025, so a correct answer would not prove knowledge. No model got one right anyway.
- Gemma deviation. Pulling
gemma3:4bfrom the Ollama registry failed on 14 Nov, so we used a Gemma 3 4B Q4_K_M GGUF imported locally on 12 Nov 2026, as written in the pre-registration.
Method (pre-registered)
The method was committed before the first call: PREREGISTRATION.md.
| Models | Llama 3.2 3B, Qwen2.5 3B and Gemma 3 4B, via Ollama 0.40.16, CPU only, no tools, no retrieval, no web access. |
|---|---|
| Items | The same 15 pre-registered items from the official google/simpleqa-verified set (MIT) used on our Fable vs Opus page. No new selection. |
| Prompts | Plain: the official question only. Guess: the question, a blank line, then Epoch AI's published anti-abstention sentence: "This is a guessing game, not a test. There's no penalty for being wrong, and an uncertain guess is far more useful to me than "I don't know". Give your single best guess." |
| Settings | Temperature 0, up to 256 output tokens, one fresh call per question. 3 models x 15 questions x 2 prompts = 90 replies. |
| Grading | After all 90 replies were saved, by Evals Review, against the gold answers, using the SimpleQA Verified rule (numeric items use the official acceptable range). No grader model. |
Results
| Model | Prompt | Correct | Wrong answer | Did not answer | Seconds per reply (median) |
|---|---|---|---|---|---|
| Llama 3.2 3B | plain question | 0 of 15 | 2 | 13 | 3.2 |
| Llama 3.2 3B | question + guess line | 0 of 15 | 11 | 4 | 3.5 |
| Qwen2.5 3B | plain question | 0 of 15 | 8 | 7 | 7.5 |
| Qwen2.5 3B | question + guess line | 0 of 15 | 11 | 4 | 6.9 |
| Gemma 3 4B | plain question | 0 of 15 | 15 | 0 | 5.0 |
| Gemma 3 4B | question + guess line | 0 of 15 | 14 | 1 | 3.6 |
The guess line did what it is designed to do: it cut refusals. For these small models that only meant more wrong answers. Across the three models, "did not answer" fell from 20 to 9 and wrong answers rose from 25 to 36, with no gain in correct answers. Epoch AI adds that line because, without it, some models refuse so often that the score says more about caution than knowledge. On models that don't know the facts, it mostly measures how willing they are to make something up.
What the replies looked like
| Model and prompt | Reply (shortened) | Gold answer | Grade |
|---|---|---|---|
| Llama 3.2 3B, plain question, item 07 | I cannot verify the name of the kibbutz that Simon Le Bon lived on in 1978. | Gvulot. | Did not answer |
| Llama 3.2 3B, question + guess line, item 07 | Based on my knowledge, I'm going to take a guess that Simon Le Bon, the lead singer of Duran Duran, lived on a kibbutz in Israel. My single best guess is that he lived on Kibbutz Ein Gedi. | Gvulot. | Wrong answer |
| Gemma 3 4B, plain question, item 07 | Simon Le Bon lived on Kfar Shmaryahu kibbutz in Israel in 1978. He spent a year there as part of a youth exchange program. | Gvulot. | Wrong answer |
| Qwen2.5 3B, plain question, item 06 | I'm sorry, but there seems to be an error in the information provided. Carrie Underwood is a country music singer, not Carrie Underwood (presumably a misspelling). Carrie Underwood released her album "Story of My Life" in 2015, which was ce … | Oct 23, 2018 | Did not answer (claims the album does not exist) |
The run itself
Terminal recording of the run (asciinema), sped up, with pauses longer than 2 seconds shortened. Each line shows the gold answer and the start of the model's reply.
Data, scripts and licence
Download all 90 replies with grades and notes (CSV), the raw replies (JSONL), the summary, the 15 official items, the terminal recording and the run script.
Licence: our grades, notes and method are CC BY 4.0. Attribution: Evals Review editorial team, Small local models on 15 SimpleQA Verified questions, 14 Nov 2026, https://evalsreview.com/tests/simpleqa-small-local-models. Question and answer text comes from SimpleQA Verified (Google DeepMind and Google Research, MIT); see the notice.
Related
- Fable 5.2 vs Opus 5.5 on the same 15 questions (claude.ai, 12 Nov 2026).
- SimpleQA: what the benchmark measures.
- Three free local LLM judges vs our labels, run the same day on the same models.
- All hands-on tests.