# Pre-registration: which free local judge agrees with our reference labels?

Written 14 Nov 2026 IST, before any judge call in this test. Approved by the site owner 14 Nov 2026, 3:07 PM IST ("Run the local judges vs human grades test"). Cost: $0 (local models on the box CPU, no API keys, no sign-ins).

## Question
On the same answers used in /tests/judge-consistency, how often do three small open models, used as PASS/FAIL judges with one fixed rubric, agree with our reference labels, and how often do they give the same verdict when asked again?

## Items and labels
- Items: `data/items.jsonl` (30 answers = 10 questions x good/mid/bad; CC BY 4.0, ours; same file as test #3).
- Labels: `data/labels.csv`, written 11 Nov 2026 for test L1-03 before any judge saw these items. 8 mid answers are marked ambiguous and are **excluded**. Official set = the 22 clear items (10 PASS, 12 FAIL).
- The labels are Evals Review's own reference labels, not an independent panel.

## Judges (Ollama 0.40.16, CPU only)
- `llama3.2:3b` (Ollama registry, id a80c4f17acd5)
- `qwen2.5:3b` (Ollama registry, id 357c53fb659c)
- Gemma 3 4B Q4_K_M: the registry pull of `gemma3:4b` failed on 14 Nov (download redirect blocked on the box), so we use the local GGUF import `gemma3-4b-local` (id 6ca894d89220, blob sha256-04a43a22e8d2...), created 12 Nov 2026 on this box. Recorded as a deviation.

## Prompt
`data/rubric.txt` (the L1-03 rubric), then `Question:` and `Answer to grade:` blocks, exactly as in the L1-03 prompt files. One fresh /api/generate call per verdict, no chat history.

## Settings
Ollama defaults for sampling (temperature as reported by `ollama show`), no seed, num_predict 200, one model loaded at a time. Order: model by model (llama, qwen, gemma); inside a model, run 1 over the 22 items in labels.csv order, then run 2, then run 3. 3 models x 22 items x 3 runs = 198 calls.

## Parsing
Case-insensitive `VERDICT:\s*(PASS|FAIL)`. Both or neither present -> `unparsed`, counted as a disagreement.

## Metrics (all fixed now)
1. Primary: majority-vote agreement with the label over the 22 items (majority of the 3 parsed verdicts; no majority -> disagreement).
2. Per-call agreement out of 66.
3. False PASS on the 12 FAIL items and false FAIL on the 10 PASS items (per call).
4. Stable items: all 3 runs gave the same parsed verdict.
5. Unparsed replies.
6. Wall-clock seconds per verdict (median, p90), first call per model excluded as load time.

## What we will not claim
No ranking of these models as judges in general. 22 items, one rubric, one box. Wilson 95% intervals are reported for the primary metric.
