Hands-on tests of LLM eval tools
Tests we run ourselves, on one machine. Most use a free local model and no paid API. One model check uses the claude.ai consumer app on an existing plan instead. Two video-generation pages compare published public tables or run an official public scorer, with no paid API. Each page gives the question, the method, the setup, the raw results and the limits, plus scripts so you can rerun it.
Tests
- Quickstart reality check (11 Nov 2026): Five official quickstarts (Promptfoo, DeepEval, Ragas, Inspect AI, lm-evaluation-harness) followed as written with a local model: steps, breaks, fixes and time to first result.
- LLM-as-judge consistency (11 Nov 2026): The same answers graded 5 times by one local judge model through DeepEval G-Eval, Promptfoo llm-rubric and a plain prompt, at two temperatures, plus answer-order swaps and a reworded rubric.
- Fable 5.2 vs Opus 5.5 on SimpleQA Verified (12 Nov 2026): 15 pre-registered SimpleQA Verified questions, run once each in the claude.ai consumer app (existing plan, no extra spend): both models answered all 15 correctly; Opus searched the web on 10 items, Fable on 6. Too small to confirm or refute Epoch AI's official scores.
- Three free local LLM judges vs our labels (14 Nov 2026): Llama 3.2 3B, Qwen2.5 3B and Gemma 3 4B, running locally through Ollama, graded 22 labelled answers PASS or FAIL, 3 times each. By majority vote they matched our labels on 21, 21 and 20 of 22, but each one repeated the same confident mistakes.
- Small local models on SimpleQA Verified (14 Nov 2026): Llama 3.2 3B, Qwen2.5 3B and Gemma 3 4B, local through Ollama with no web access, on the 15 pre-registered SimpleQA Verified items, plain vs Epoch AI's guess prompt, hand-graded: 0 of 90 answers were correct.
- Tracing overhead: Phoenix vs MLflow, both local (14 Nov 2026): 200 local LLM calls per setup with no tracing, Arize Phoenix 21.0.0, or MLflow 3.18.0, plus a 2,000-call traced no-op burst, all on one CPU box with local servers.
- Human video Elo vs physics scores (14 Nov 2026): Published AA-Video-T2V v2.1, Physics-IQ, Physics-IQ Verified and VideoPhy-2 tables compared on 14 Nov 2026. After matching only the same version and input type, no pair of boards shared four models, so no correlation was computed. We did not run any video model.
- Official Physics-IQ scorer on a public subset (14 Nov 2026): Official Apache-2.0 scorer run on a CPU on 3 official original scenarios and 3 views. Two real takes scored 100.0 on that 9-clip subset. A published model row could not be re-scored: official Verified ground truth is gated.