# Independent test #3: LLM-as-judge consistency

**Question.** If you ask an LLM judge to grade the same answer several times, does it give the same score? And does
rubric wording or answer order change it?

Run on 11 Nov 2026 (IST) by the Evals Review editorial team. Zero spend: one local open model via Ollama, no hosted APIs.
Draft page: `/tests/judge-consistency` (app/tests/judge-consistency/page.tsx). Every number on the page comes from
`results/summary.json`, produced by `scripts/analyze.py` from the raw rows in `results/*.jsonl`.

## Setup
- **Judge model:** `llama3.2:3b` (official Ollama library tag; digest and Ollama version in `results/versions.txt`).
  Llama 3.2 Community License: using the model and publishing results is allowed; we do not redistribute it.
  We did not use `qwen2.5:3b` (test #1's model) because Qwen2.5-3B is under the Qwen Research License (non-commercial).
- **Machine:** see `results/versions.txt` (CPU only, no GPU).
- **Frameworks (built-in judge setups, defaults kept):**
  - DeepEval `GEval` metric with `deepeval.models.OllamaModel` (+ the `ollama` Python client DeepEval asks for).
    A fresh `GEval` object per repeat, so its auto-generated evaluation steps are regenerated each time, as in a
    fresh run. Verdict: DeepEval's default threshold 0.5 (`metric.success`). DeepEval's OllamaModel sends
    temperature 0 unless told otherwise; we pass 0.8 or 0 explicitly.
  - Promptfoo `llm-rubric` assertion, grader `ollama:chat:llama3.2:3b`, `echo` provider so the "output" is our fixed
    answer. Run with `--no-cache` and `PROMPTFOO_CACHE_ENABLED=false` (Promptfoo caches grader calls by default).
    Verdict: the grader's own `pass`. Score: the grader's own score (0-1; a score above 1 is read as out of 10).
  - Plain prompt baseline: one `/api/chat` call asking for a whole number 1-10. Verdict: score >= 6.
    "Default temperature" for the baseline = no temperature sent, i.e. Ollama's default (0.8).
- **Rubric v1** (all three methods, each wrapped in the framework's own prompt) and **rubric v2** (same intent, other
  words; plain prompt only). Exact texts in `scripts/common.py`, `scripts/deepeval_geval.py`, `scripts/promptfoo_rubric.py`.
- **Dataset:** `data/items.jsonl`, 30 short answers written by us: 10 questions x {good, mid, bad}. The tier is our
  intended quality label, not an independent human rating. Licence: CC BY 4.0 (ours).

## Runs
| Run | What | Repeats |
|---|---|---|
| raw v1 | plain prompt, rubric v1, temp default (0.8) and 0 | N=5 each |
| deepeval | G-Eval, temp 0.8 and 0, on a 12-item subset (q01, q04, q07, q10 x good/mid/bad): one G-Eval measure took ~55-67 s on this CPU with the other streams running, so all 30 items x 5 x 2 would not finish before 10 AM (cut at 07:20 IST) | N=5 each |
| promptfoo | llm-rubric, temp 0.8 and 0 | N=5 each (one `promptfoo eval` per repeat) |
| pairwise | plain prompt; per question, good vs mid and good vs bad, shown A/B and B/A | temp 0: 1; temp 0.8: 3 |
| raw v2 | plain prompt, reworded rubric | N=3 at temp 0, then N=3 at default |

Repeats run in the order repeat -> temperature -> item, so if the time limit (`JC_DEADLINE`, 08:50 IST) cuts a run,
every finished repeat covers both temperatures. `analyze.py` keeps only repeats finished for all 30 items.
The actual repeat counts reached are in `results/summary.json` (`repeats_per_item`) and on the page.

## Metrics
- **Same score every run:** share of items whose N scores are all identical.
- **Mean score std dev:** per-item population std dev of the 0-10 score, averaged over items.
- **Verdict flipped:** share of items whose pass/fail differs between at least two of the N repeats.
- **Matches our labels:** share of runs where a good answer passed or a bad answer failed (mid answers excluded).
- **Position swap:** for each pair and repeat, did the judge pick the good answer in one order and not in the other?
- **Rubric rewording:** per item, most common score under v1 vs v2 at the same temperature.
- Unparsed outputs and framework errors are counted (`unparsed_or_errors`), never dropped silently.

## Timing and CPU sharing (test #1)
Test #1 (quickstart reality check) measures wall-clock install/run times on the same box. `scripts/run_all.sh` waits
until test #1 writes "timed runs done" to `../quickstart-reality-check/STATUS.md`, or until 08:15 IST (then runs at
one parallel slot and says so). What actually happened is in `logs/start_reason.txt` and `logs/run_all.log`.
Judge runs use a separate Ollama server on port 11435 (`OLLAMA_NUM_PARALLEL` recorded in `logs/start_reason.txt`),
so test #1's server is never touched. The four run streams share that server, so requests can be batched together;
llama.cpp does not promise bit-identical output across batch sizes, so some temperature-0 variation may come from
batching, as it would on any shared inference server.

Model download: in this sandbox `ollama pull` fails (its proxy makes Ollama reject the registry's CDN redirect), so
`scripts/fetch_model.sh` downloads the same manifest and blobs from registry.ollama.ai with curl and checks every
sha256 before installing them. On a normal machine `ollama pull llama3.2:3b` does the same.

## Rerun
```
cd tests/judge-consistency
JC_NOWAIT=1 bash scripts/run_all.sh          # needs ollama, python3, node; ~1-2 h on an 8-core CPU
# or step by step (Ollama on :11435 or set JUDGE_OLLAMA_URL):
python scripts/raw_judge.py --rubrics v1 --temps default,0 --n 5
python scripts/deepeval_geval.py --temps 0.8,0 --n 5
python scripts/promptfoo_rubric.py --temps 0.8,0 --n 5
python scripts/pairwise.py --temps 0,default --n0 1 --nd 3
python scripts/raw_judge.py --rubrics v2 --temps 0,default --n 3
python scripts/analyze.py
```
Scripts are resumable: they skip rows already in `results/*.jsonl`. Delete those files for a clean rerun.
Schedule: weekly series; this test is rerun quarterly with current versions.

## Files
- `data/items.jsonl` - the 30 items
- `scripts/` - all code; `promptfooconfig-*.json` are the generated Promptfoo configs
- `results/raw.jsonl`, `deepeval.jsonl`, `promptfoo.jsonl`, `pairwise.jsonl` - one row per judge call, with the raw reply
- `results/promptfoo-raw/` - Promptfoo's own output files
- `results/summary.json`, `results/all-scores.csv` - computed metrics and a flat CSV
- `results/versions.txt` - model digest, tool versions, machine, date
- `logs/` - run logs

## Limits (also on the page)
One small local model on CPU, not a frontier judge; not a framework ranking (each framework's prompt, parsing and
pass rule differ); our own 30 short items and labels; temperature 0 is not a determinism guarantee.

## What actually happened on 11 Nov 2026 (run log)
- 07:15 IST: test #1 wrote "timed runs done 07:15 IST"; judge runs started at 07:15:49 (`logs/start_reason.txt`: test1_done, 4 parallel slots).
- ~07:18: one DeepEval G-Eval measure took 55-67 s, so DeepEval was cut to a 12-item subset (q01, q04, q07, q10 at all three tiers).
- ~07:20: an environment variable from a mock smoke test (`JUDGE_N=2`) had leaked into the first launch; raw v1, DeepEval and Promptfoo
  were resumed with `--n 5`. The scripts are resumable, so finished rows were kept and no row was duplicated.
- ~07:21: DeepEval split into one process per temperature; a third pairwise set (partial vs wrong) was added because good-vs-partial and
  good-vs-wrong pairs never changed with order.
- The shared box ran short of memory several times and Ollama restarted its model process (`signal: killed` in `logs/ollama-11435.log.gz`).
  Calls in flight failed: 6 DeepEval calls (kept in `results/deepeval-errors-retried.jsonl`, rerun once at ~08:16) and 1 Promptfoo call
  (kept as an error, shown on the page).
- 08:19: all runs complete. Every v1 group reached 5 repeats; raw v2 has 3. `scripts/verify_summary.py` re-counts the headline numbers
  from the raw JSONL independently of `analyze.py` (`logs/verify_summary.log`: ALL OK).
- 08:20: while shutting down the judge server, the test #1 Ollama server on :11434 was also stopped by mistake and restarted within ~20 s
  (after test #1's timed runs, which had finished at 07:15).

## Licence
The dataset (`data/items.jsonl`), raw results, computed summaries and scripts in this folder are published under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). See `LICENSE`.

Attribution: Evals Review editorial team, LLM-as-judge consistency test, 11 Nov 2026, https://evalsreview.com/tests/judge-consistency

The judge model is Llama 3.2 3B under the Llama 3.2 Community License. We used it and publish our scores; we do not redistribute the model weights.
