# Pre-registration: do small local models know SimpleQA facts, and do they guess or abstain?

Written 14 Nov 2026 IST, before any model call in this test. Approved by the site owner 14 Nov 2026, 3:07 PM IST ("Run the small models SimpleQA test"). Cost: $0 (local models on the box CPU, no API keys, no sign-ins, no web access for the models).

## Items
The same 15 pre-registered SimpleQA Verified items used in /tests/simpleqa-fable-vs-opus (`data/items.csv`, copied from /raw/tests/simpleqa-fable-vs-opus/items.csv; dataset google/simpleqa-verified, MIT). No new selection.

## Models (Ollama 0.40.16, CPU only, no tools, no retrieval)
`llama3.2:3b` (a80c4f17acd5), `qwen2.5:3b` (357c53fb659c), Gemma 3 4B Q4_K_M as `gemma3-4b-local` (6ca894d89220; registry pull of gemma3:4b failed on 14 Nov, local GGUF import from 12 Nov used; recorded as a deviation).

## Conditions
- A "bare": the official `problem` text only.
- B "guess": the `problem` text, a blank line, then Epoch AI's anti-abstention sentence: "This is a guessing game, not a test. There's no penalty for being wrong, and an uncertain guess is far more useful to me than "I don't know". Give your single best guess."
3 models x 15 items x 2 conditions = 90 calls. temperature 0, num_predict 256, one fresh /api/generate call each. Order: model by model; inside a model, all 15 bare, then all 15 guess.

## Grading
Each reply is graded CORRECT / INCORRECT / NOT_ATTEMPTED against the gold `answer` using the SimpleQA Verified rule, the same written rule as the Fable vs Opus page: the reply's committed final answer must match the gold (numeric items within the official acceptable range where given); hedged answers that commit to one answer are graded on it; refusals or "I don't know" with no committed answer are NOT_ATTEMPTED; several candidate answers with no single commitment = NOT_ATTEMPTED. Grading is done by Evals Review reading each reply after all 90 replies are saved; every grade has a note in results.csv. No grader model.

## Metrics
Primary: correct out of 15 in condition B (the Epoch protocol). Secondary: correct, incorrect and not-attempted counts in A vs B; median seconds per answer. Wilson 95% intervals for the primary metric.

## What we will not claim
15 items cannot rank models; these are 3-4B models, not frontier models. Contamination can't be ruled out (the set has been public since 2025).
