LLM-as-judge consistency: does the same answer get the same score?
Short answer: Not reliably. With one small local judge (Llama 3.2 3B), the same answer often got a different verdict on a rerun with nothing changed: at Ollama's default temperature, 30% of answers flipped between pass and fail with a plain prompt, 36.7% with Promptfoo's llm-rubric and 58.3% with DeepEval's G-Eval. Temperature 0 cut the plain prompt's flips to 3.3%, but the framework judges still flipped on 20% (Promptfoo) and 16.7% (DeepEval) of answers, and swapping the order of a partial and a wrong answer changed the winner in 14 of 30 runs.
Why this matters if you are picking an eval tool
Most eval tools now ship an LLM judge: DeepEval has G-Eval, Promptfoo has llm-rubric, and many teams write their own grading prompt. If the judge gives the same answer a different score on a rerun, a "regression" in your dashboard may be noise, and a pass/fail gate in CI may flip with no code change. Before you trust a judge score, you need to know how much it moves when nothing changes.
Setup
| Judge model | Llama 3.2 3B (llama3.2:3b) via Ollama 0.40.16, digest a80c4f17acd5 (Llama 3.2 Community License). One small local model, not a frontier judge. |
|---|---|
| Frameworks | DeepEval 4.3.3 (review), G-Eval metric with the OllamaModel class; Promptfoo 0.125.1 (review), llm-rubric assertion with an Ollama grader; plus a plain prompt sent straight to Ollama. |
| Machine | Shared 8-core cloud Linux box, CPU only (Intel(R) Xeon(R) Processor), 15 GB; no GPU |
| Dataset | 30 short answers we wrote: 10 questions (facts, Python, SQL, Git, science), each with a good, a partial and a wrong answer. The quality tier is our label, not an independent rater's. Published in full with the results. |
| Repeats | Each answer graded 5 times per method at temperature 0.8 (Ollama's default) and 5 times at temperature 0. DeepEval ran on 12 of the 30 items (4 questions at all 3 quality levels), 5 runs per item at each temperature, because one G-Eval call took about 30 to 75 seconds on this CPU and the full set would not finish in the time we had. |
| Verdict rule | Each framework's own: DeepEval passes at a score of 0.5 or more (its default threshold); Promptfoo uses the grader's own pass/fail; the plain prompt passes at 6 of 10 or more. Scores are shown on a 0-10 scale. |
| Extra checks | Position swap: for each question, the good answer against the partial and against the wrong one, shown as A/B and again as B/A. Rubric rewording: the plain prompt with a second rubric that says the same thing in other words. |
| Date | 11 Nov 2026, IST. Judge runs started at 07:15 IST, right after test #1 wrote that its timed runs were done, so they did not affect test #1's timings. Other jobs shared the CPU during our runs, so per-call times here are not speed benchmarks. |
What we found
| Method | Temp | Items | Same score every run | Mean score std dev (0-10) | Verdict flipped | Matches our labels* | Mean score: good / partial / wrong | Unparsed / errors |
|---|---|---|---|---|---|---|---|---|
| Plain prompt (1-10) | 0.8 (Ollama default) | 30 | 13.3% | 0.68 | 30% (9 of 30) | 94% | 7.96 / 4.72 / 3.68 | 0 of 150 |
| Plain prompt (1-10) | 0 | 30 | 93.3% | 0.04 | 3.3% (1 of 30) | 95% | 7.9 / 4.76 / 3.68 | 0 of 150 |
| DeepEval G-Eval | 0.8 | 12 | 8.3% | 1.37 | 58.3% (7 of 12) | 80% | 6.2 / 2.85 / 2.6 | 0 of 60 |
| DeepEval G-Eval | 0 | 12 | 58.3% | 0.43 | 16.7% (2 of 12) | 95% | 6.85 / 3.05 / 2.1 | 0 of 60 |
| Promptfoo llm-rubric | 0.8 | 30 | 36.7% | 1.93 | 36.7% (11 of 30) | 70% | 4.48 / 0.33 / 0.73 | 0 of 150 |
| Promptfoo llm-rubric | 0 | 30 | 66.7% | 0.99 | 20% (6 of 30) | 63% | 2.89 / 0.47 / 0.5 | 1 of 150 |
One answer, graded again and again
Question: "What is the boiling point of water at sea level in Celsius and Fahrenheit?" Answer (our label: wrong): "Water boils at 90 degrees Celsius, which is 180 degrees Fahrenheit."
| Method | Temp | Scores |
|---|---|---|
| Plain prompt | 0.8 | 4, 8, 4, 4, 7 |
| Plain prompt | 0 | 4, 4, 4, 4, 4 |
| Promptfoo | 0.8 | 0, 0, 0, 0, 0 |
| Promptfoo | 0 | 0, 0, 0, 0, 0 |
| DeepEval | 0.8 | not run |
| DeepEval | 0 | not run |
| Pair | Temp | Pair runs | Winner changed when order swapped | Picked the better answer | Picked slot A |
|---|---|---|---|---|---|
| Good vs wrong | 0 | 10 | 0% (0 of 10) | 100% | 50% |
| Good vs wrong | 0.8 (Ollama default) | 30 | 0% (0 of 30) | 100% | 50% |
| Good vs partial | 0 | 10 | 0% (0 of 10) | 100% | 50% |
| Good vs partial | 0.8 (Ollama default) | 30 | 0% (0 of 30) | 100% | 50% |
| Partial vs wrong | 0 | 10 | 30% (3 of 10) | 75% | 35% |
| Partial vs wrong | 0.8 (Ollama default) | 30 | 46.7% (14 of 30) | 66.7% | 36.7% |
| Temp | Items | Score changed | Mean change (v2 minus v1) | Verdict changed |
|---|---|---|---|---|
| 0 | 30 | 23.3% | -0.3 | 6.7% (2 items) |
| 0.8 (Ollama default) | 30 | 53.3% | -0.33 | 13.3% (4 items) |
What it means
- Temperature mattered most for the plain prompt. At 0.8: 13.3% of answers got the same score on all 5 runs and 30% flipped verdict. At 0: 93.3% same score, 3.3% flipped. Of the 9 flips at 0.8, 5 were partial answers (where either verdict is defensible) and 4 were wrong answers that passed at least once.
- The framework judges moved more, even at temperature 0. Both ask the judge for a written reason with the score, and the plain prompt asked for the number only; longer outputs give small differences more room to change the result. That is a likely cause, not one we isolated. DeepEval's G-Eval also writes its own evaluation steps for each new metric, and those steps differed between runs for 24 of 24 item-temperature combinations; passing your own evaluation_steps removes that source.
- Steady is not the same as right. Promptfoo's grader on this model was strict: good answers averaged 2.89 of 10 at temperature 0, and its verdicts matched our labels 63% of the time, against 95% for the plain prompt and 95% for G-Eval on its 12-item subset. Check any judge against cases you have labelled before you trust it.
- Order bias showed up when the two answers were close. Good answer against a partial or wrong one: the judge picked the good one in both orders every time. Partial against wrong: the winner changed with the order in 14 of 30 runs at 0.8 and 3 of 10 at 0, and slot A won only 36.7% of the time at 0.8.
- Wording mattered too. A rubric that says the same thing in other words changed the most common score for 23.3% of answers at temperature 0 and flipped the verdict for 2 of 30.
What to do with your own judge: run each case at least 3 times and look at the spread before you call a change a regression; set temperature 0 for grading; keep the rubric text fixed and versioned; for pairwise grading, run both orders and count a win only when both agree; and check the judge against 50 to 100 cases you have labelled yourself. See how to write AI evals.
Limits: read before you quote a number
- One small local model. A 3-billion-parameter model on a CPU is not GPT-class or Claude-class. Frontier judges may be far more consistent; we did not run them here. These numbers describe this model, not LLM judges in general.
- Not a framework ranking. Each framework wraps the judge in its own prompt, output format and pass rule, and we kept those defaults. A difference between rows mixes all of that, on 30 items.
- Our dataset, our labels. 30 short answers written by us at three quality levels; real eval cases are longer and fuzzier, which usually makes judges less consistent, not more.
- Temperature 0 is not a guarantee. Temperature 0 did not make any of the three judges fully repeatable here. Our run streams shared one Ollama server with 4 parallel slots, and llama.cpp does not promise identical output across batch sizes, so batching is a likely cause; hosted APIs that batch requests can behave the same way.
- 1 judge call failed with a server error (the shared box ran short of memory and the model server restarted). They are counted in the Unparsed / errors column and left out of the consistency figures, not scored as fails.
- 6 DeepEval calls hit the same server error; we kept the failed rows in the published files and reran those 6 calls once.
- The position-swap and rewording checks used only the plain prompt, not the frameworks.
Rerun it
Everything is published with this page: the full folder (the 30 items, scripts, every raw judge response as JSONL, versions and logs), all scores as CSV, the computed summary, the method README and the CC BY 4.0 licence. On any machine with Ollama, Python 3 and Node:
tar xzf judge-consistency.tar.gz && cd judge-consistency
JC_NOWAIT=1 bash scripts/run_all.sh # pulls Llama 3.2 3B (llama3.2:3b), installs DeepEval + Promptfoo, runs, writes results/summary.jsonLicence: the 30-item dataset, raw results, summaries and scripts are published under CC BY 4.0. Attribution: Evals Review editorial team, LLM-as-judge consistency test, 11 Nov 2026, https://evalsreview.com/tests/judge-consistency.
We plan to rerun this test every quarter with the newest versions, and to add a larger public dataset and hosted judges when budget allows.