Three free local LLM judges vs our labels
Short answer: On 14 Nov 2026 we asked three small open models, running on a CPU through Ollama, to grade 22 labelled answers PASS or FAIL, 3 times each. By majority vote, Llama 3.2 3B and Qwen2.5 3B matched our labels on 21 of 22 answers and Gemma 3 4B on 20 of 22. The misses were not random: every majority-vote miss was the same wrong verdict on all 3 runs, so repeating the call did not catch it. Gemma passed "water boils at 90 degrees Celsius" all 3 times.
Limits: read these before quoting a number
- 22 answers. A 95% interval on 21 of 22 runs from about 78.2% to 99.2%. These three models are not ranked against each other here; their scores overlap.
- The labels are ours. We wrote the PASS/FAIL labels on 11 Nov 2026, before any judge saw these items, for an earlier test. They are not an independent panel. 8 borderline answers were dropped as ambiguous before this run.
- Small models on a CPU. 3 to 4 billion parameters, 4-bit quantised. Hosted frontier judges may do much better; we did not run them.
- One rubric, one prompt. A different rubric or a few-shot prompt could change the results.
- Gemma deviation. Pulling
gemma3:4bfrom the Ollama registry failed on 14 Nov (the download redirect was blocked on our machine), so we used a Gemma 3 4B Q4_K_M GGUF imported locally on 12 Nov 2026. This was written into the pre-registration before the run.
Method (pre-registered)
The method was committed to our repo before the first call: PREREGISTRATION.md.
| Judges | Llama 3.2 3B (llama3.2:3b), Qwen2.5 3B (qwen2.5:3b), Gemma 3 4B Q4_K_M (local import), via Ollama 0.40.16, CPU only (8 vCPU, 16 GB RAM). |
|---|---|
| Answers | 22 clear answers from our judge consistency test: 10 labelled PASS, 12 labelled FAIL, across 10 questions (HTTP codes, Python, Git, physics, ML terms). |
| Prompt | A strict PASS/FAIL rubric, the question and the answer. The judge must reply VERDICT: PASS or VERDICT: FAIL plus one reason. Fresh call each time, no chat history. |
| Repeats | 3 runs per answer per judge, at Ollama's default sampling (no temperature set, no seed): 198 verdicts. |
| Scoring | Primary: majority of the 3 verdicts vs the label. Also per-call agreement, false PASS on the 12 FAIL answers, false FAIL on the 10 PASS answers, and answers where all 3 runs agreed. |
Results
| Judge | Majority vote matches label | Single calls matching | Passed a wrong answer | Failed a right answer | Same verdict all 3 runs | Seconds per verdict (median) |
|---|---|---|---|---|---|---|
| Llama 3.2 3B | 21 of 22 | 63 of 66 | 0 of 36 calls | 3 of 30 calls | 22 of 22 | 3.2 |
| Qwen2.5 3B | 21 of 22 | 61 of 66 | 3 of 36 calls | 2 of 30 calls | 20 of 22 | 3.5 |
| Gemma 3 4B | 20 of 22 | 60 of 66 | 6 of 36 calls | 0 of 30 calls | 22 of 22 | 4.0 |
No reply was unparseable: every one of the 198 replies contained exactly one verdict. Llama 3.2 3B never passed a wrong answer but failed one right answer on all 3 runs. Gemma 3 4B never failed a right answer but passed two wrong ones on all 3 runs. Qwen2.5 3B made both kinds of mistake and was the only judge that changed its verdict between runs (on 2 answers).
The mistakes, word for word
| Answer being graded | Judge and verdicts | Judge's reason (run 1) | Our label |
|---|---|---|---|
| What is the boiling point of water at sea level in Celsius and Fahrenheit? "Water boils at 90 degrees Celsius, which is 180 degrees Fahrenheit." | Gemma 3 4B: PASS, PASS, PASS | The answer accurately states the boiling point of water at sea level in both Celsius and Fahrenheit, providing both values correctly. | FAIL |
| How do I undo my last Git commit but keep the changes in my working files? "Run git reset --hard HEAD~1. Your changes will stay in your files." | Qwen2.5 3B: PASS, PASS, PASS | The answer is factually correct and covers the key points of the question, suggesting the appropriate command to undo the last Git commit while keeping changes in the working directory. | FAIL |
| How do I undo my last Git commit but keep the changes in my working files? "Run git reset --hard HEAD~1. Your changes will stay in your files." | Gemma 3 4B: PASS, PASS, PASS | The answer correctly identifies the `git reset --hard HEAD~1` command as the method to undo the last commit while preserving the changes in the working directory. | FAIL |
| What does HTTP status code 404 mean? "404 Not Found means the server was reached but it has no resource at the requested URL. It is a client-side error class (4xx); the page may have been moved or deleted, or the URL may be mistyped." | Llama 3.2 3B: FAIL, FAIL, FAIL | The answer incorrectly states that 404 is a client-side error class (4xx), when in fact, 404 is a server-side error class (5xx). | PASS |
So a small local judge can be steady and still wrong. Running the same call 3 times shows how noisy a judge is; it does not show whether the judge is right. For that you need a small set of answers you have labelled yourself.
The run itself
Terminal recording of the run (asciinema), sped up, with pauses longer than 2 seconds shortened. Every line is one verdict.
Data, scripts and licence
Download all 198 verdicts (CSV), the raw replies (JSONL), the summary, our labels with reasons, the answers, the terminal recording and the run script. The scripts are also in our repo under tests/local-judges-vs-labels/.
Licence: answers, labels, verdict files and method notes are CC BY 4.0. Attribution: Evals Review editorial team, Three free local LLM judges vs our labels, 14 Nov 2026, https://evalsreview.com/tests/local-judges-vs-labels. Model outputs come from Llama 3.2 (Llama 3.2 Community License), Qwen2.5 3B (Qwen Research License) and Gemma 3 (Gemma Terms of Use).
Related
- Does an LLM judge give the same score twice? The same answers, scored 1 to 10 through DeepEval, Promptfoo and a plain prompt.
- LLM-as-judge tools and our eval checklist.
- DeepEval and Promptfoo reviews.
- All hands-on tests.