LLM eval checklist: 15 steps, with the tools that support each
Short answer: a solid LLM eval checklist starts from real failures, keeps a versioned test set, runs code checks before LLM judges, checks the judge against human labels, puts error bars on scores, gates deploys in CI and keeps scoring production traffic. Below, each of the 15 steps lists the tools whose own docs show support, and says so where we found none.
TL;DR
- Steps 1 to 4 build the test set: real failures, versioned data, deliberate synthetic cases, code checks first.
- Steps 5 to 11 make scores trustworthy: split RAG scoring, check agent paths, use pass^k, simulate users, calibrate and de-bias judges, add error bars.
- Steps 12 to 15 keep it running: CI gates, production evals, cost and latency, and feeding failures back.
- Want practice? Learn by doing gives one small project per step.
Checklist at a glance
| # | Step | Tools with documented support |
|---|---|---|
| 1 | Read failures first | Langfuse, LangSmith, Braintrust (annotation) |
| 2 | Versioned golden dataset | LangSmith, Braintrust, Langfuse, Opik, Weave |
| 3 | Deliberate synthetic data | DeepEval Synthesizer |
| 4 | Code checks first | Promptfoo, Ragas |
| 5 | Retrieval vs generation | Ragas |
| 6 | Agent trajectories | Promptfoo, DeepEval, Ragas |
| 7 | pass^k | none found; compute it yourself |
| 8 | Simulated users | DeepEval |
| 9 | Judge as classifier | Langfuse (calibration) |
| 10 | Judge bias | LangSmith (randomize_order) |
| 11 | Error bars | none found; maths only |
| 12 | CI gates | Promptfoo, DeepEval, LangSmith, Braintrust, Langfuse, Arize, Ragas |
| 13 | Online evals | Langfuse, Arize |
| 14 | Cost and latency | Langfuse, Promptfoo |
| 15 | Close the loop | any versioned dataset + annotation |
1. Read failures before picking metrics
Why: Metrics chosen up front measure what you guessed would break, not what does.
How: Pull a sample of real traces, write a free-text note on every bad one (open coding), then group the notes into a short list of failure types (axial coding) and count them. The biggest groups become your first metrics.
Tools: Annotation queues to label traces: Langfuse [Langfuse pricing (annotation queues)], LangSmith [LangSmith docs], Braintrust human review [Braintrust docs]. The grouping itself is a manual step; we found no tool doc that automates axial coding.
2. Keep a versioned golden dataset
Why: If the test set changes silently, last month's score and today's aren't comparable.
How: Store each case with its input, the expected answer or rubric, and tags such as intent or difficulty. Over-sample rare but costly cases, and pin a dataset version next to each prompt version.
Tools: Dataset versions: LangSmith (versions and tags) [LangSmith docs], Braintrust (pin versions with snapshots) [Braintrust docs], Langfuse [Langfuse docs], Opik [Opik docs], W&B Weave [W&B Weave docs].
3. Generate synthetic cases on purpose
Why: Asking a model for "50 test questions" returns 50 versions of the same question.
How: List the axes that matter (who is asking, what they want, how hard, which edge case), build combinations of those values, then turn each combination into a realistic input. Read and edit the output before it joins the golden set.
Tools: DeepEval's Synthesizer generates goldens from documents, contexts, scratch or existing goldens, with configurable evolutions and styling (task, scenario); its docs advise inspecting the output by hand [DeepEval docs]. You still define the dimensions yourself.
4. Run code checks before any judge
Why: A regex or schema check is free, instant and gives the same answer every time.
How: Validate JSON against a schema, check required fields and lengths, run generated code or SQL. Use a model grader only for what code can't decide.
Tools: Promptfoo ships deterministic assertions including is-json with schema, regex, equals, contains, is-sql and custom JavaScript or Python checks [Promptfoo docs]. Ragas also lists non-LLM metrics such as exact match and string presence [Ragas docs].
5. Score retrieval and generation separately
Why: A single answer score can't tell you whether search or writing failed.
How: Grade the retrieved chunks against labelled relevant ones, then grade the answer for faithfulness to what was retrieved.
Tools: Ragas lists retrieval metrics (context precision, context recall) apart from generation metrics (faithfulness) [Ragas docs]. See Ragas vs DeepEval and RAG evals.
6. Evaluate the agent's path, not just its answer
Why: An agent can reach the right answer with three times the tool calls it needed.
How: Log each step's tool, arguments and result. Compare the calls against a reference path, check arguments, and count steps.
Tools: Promptfoo has trajectory assertions (trajectory:tool-used, trajectory:tool-args-match, trajectory:tool-sequence, trajectory:step-count) and tool-call-f1 [Promptfoo docs]. DeepEval's Tool Correctness compares tools called with expected tools, optionally checking arguments and order [DeepEval docs]. Ragas lists tool call accuracy and tool call F1 [Ragas docs]. More in agent evals.
7. Measure pass^k, not only pass@k
Why: pass@k rewards an agent that gets it right once; users need it right every time.
How: pass@k counts a case as solved if any of k runs succeed; pass^k only if all k do. If each run succeeds 80% of the time independently, all five succeed 0.8^5 ≈ 33% of the time. Repeat each case at your production temperature.
Tools: The metric comes from tau-bench. We found no tool doc for a built-in pass^k scorer; run cases k times in any harness and compute it.
8. Test multi-turn with simulated users
Why: Single-turn tests never exercise memory or state across a conversation.
How: Give a model a persona, a goal and limits, let it talk to your system until the goal is met or a turn cap hits, then grade completion, turn count and policy breaks.
Tools: DeepEval's ConversationSimulator role-plays a user from a scenario, persona and expected outcome, stopping at a turn cap (default 10) or when the outcome is reached [DeepEval docs].
9. Treat your LLM judge as a classifier
Why: An unchecked judge is a second model you are trusting blindly.
How: Hand-label a set of pass/fail cases, split them, tune the judge prompt on one part and report true-positive and true-negative rates on a held-out part. Keep verdicts binary and pin the judge model.
Tools: Langfuse supports boolean judge scores and says to calibrate judges against labelled examples, comparing judge and human scores in Score Analytics (Cohen's kappa, confusion matrix) [Langfuse docs]. LLM-judge evaluators are also in LangSmith, Braintrust, DeepEval and others (see LLM evaluation tools). Background: LLM-as-a-judge vs agent-as-a-judge.
10. Control judge bias in A/B comparisons
Why: Judges tend to favour the first answer shown and the longer one.
How: Run each pairwise comparison in both orders and keep only verdicts that agree; cap or normalise length if verbosity skews results.
Tools: LangSmith's pairwise evaluation has a randomize_order option, described as a mitigation for positional bias (off by default) [LangSmith docs]. Swapping and agreement checks you script yourself. The MT-Bench paper documents these biases.
11. Put error bars on every score
Why: Small test sets make noise look like progress.
How: At n = 100 and 80% accuracy, the normal-approximation 95% interval is 1.96 × √(0.8 × 0.2 / 100) ≈ ±7.8 points, so roughly 72% to 88%. A move from 80% to 83% sits inside that. Use bootstrap intervals or a paired test on the same cases before calling a win.
Tools: This is maths, not a product feature: we found no tool doc we could cite for built-in confidence intervals, so we list none.
12. Gate deploys in CI
Why: Prompt, model and retrieval changes ship regressions without a gate.
How: Run the suite on every change to prompts, models, tools or retrieval settings, with a threshold per failure category, and fail the build on a drop.
Tools: Promptfoo (CI/CD in the free Community version) [Promptfoo pricing (CI/CD in Community)], DeepEval [DeepEval docs], LangSmith (pytest, Vitest) [LangSmith docs], Braintrust [Braintrust docs], Langfuse [Langfuse docs], Arize [Arize docs], Ragas [Ragas docs]. Compare two: DeepEval vs Promptfoo.
13. Keep evaluating in production
Why: Real traffic drifts away from your test set.
How: Sample live traffic, score it asynchronously with the same judges, and alert per failure category. Shadow-run a new prompt or model on real inputs before switching.
Tools: Langfuse evaluation rules run judges on live observations with a sampling rate [Langfuse docs]. Arize lists online and offline evals [Arize pricing (online and offline evals)]. Tracing options: LLM observability tools. We found no tool doc for shadow deployments to cite.
14. Track cost and latency next to quality
Why: A change that holds quality and cuts cost is a win your quality score won't show.
How: Record tokens per request, cost per successful task and p50/p95 latency for every experiment.
Tools: Langfuse tracks usage and cost per LLM call, with dashboards and spend alerts [Langfuse docs]. Promptfoo has cost and latency assertions, and trace-span-duration with percentiles [Promptfoo docs]. Plan costs: Langfuse vs LangSmith cost calculator.
15. Feed every failure back into the test set
Why: A bug found in production and not added as a case will come back.
How: Turn each production failure into a labelled case in the golden dataset (step 2), and run every new model release on the full suite before debating a switch.
Tools: Any tool with versioned datasets (step 2) plus annotation (step 1) covers this.
Learn by doing: one practice project per step
The quickest way to learn evals is to build a small app that fails and run each step on it. Each project below ends in something you can show: a file, a notebook or a CI run. The project ideas follow diva's follow-up thread; the wording, structure and tool links are ours.
- 1. Read failures first: Build a small app that can fail (a docs RAG bot, a text-to-SQL bot or a tool-calling agent), read its traces by hand, and publish the failure categories you found with a count for each. Annotate in Langfuse, LangSmith or Braintrust.
- 2. Versioned golden dataset: Turn your biggest failure categories, plus some easy and some adversarial cases, into a dataset in git, and record which dataset version every prompt change was scored on. Compare dataset features in Langfuse vs LangSmith.
- 3. Deliberate synthetic data: Write your dimensions down, generate every combination with a script, have a model phrase each one, then delete the weak ones. Ship the set with a coverage matrix showing which combinations you tested (diva gives 300 cases as an example size, not a rule). DeepEval has a synthesizer.
- 4. Code checks first: Write pytest tests that validate outputs against a Pydantic model, run generated SQL against a throwaway test database, and see how many failure categories you can catch with no LLM at all. Promptfoo has deterministic assertions too.
- 5. Retrieval vs generation: Label the relevant chunks for a set of queries, score retrieval and faithfulness separately, then change one thing (chunk size or embedding model) and write down which score moved. Ragas covers both sides.
- 6. Agent trajectories: Write reference paths for a set of agent tasks, score tool choice, arguments and step count, then make one tool fail on purpose and record whether the agent recovers. See Promptfoo, DeepEval and our agent evals comparison.
- 7. pass^k: Run every case several times at production temperature, report pass@1 next to pass^k, and list the cases that flake. Those are your reliability bugs. Any harness works, including Inspect.
- 8. Simulated users: Give a simulator model a persona, a hidden goal and one stubborn constraint, cap the turns, and grade goal completion, turns taken and policy breaks. DeepEval documents a conversation simulator.
- 9. Judge as classifier: Hand-label pass/fail examples, split them, tune the judge prompt on the dev split only, and publish a notebook with the test-split confusion matrix and every prompt version you tried. Langfuse documents judge-versus-human comparison; see LLM-as-a-judge vs agent-as-a-judge.
- 10. Judge bias: Run each pairwise comparison in both orders, keep only verdicts that agree, and plot win rate against answer length to see whether your judge rewards length. LangSmith documents randomizing the order.
- 11. Error bars: Re-score two prompt versions on the same cases, add bootstrap intervals and a paired test, and write down whether the difference survives. If it doesn't, call it noise.
- 12. CI gates: Open a pull request that brings back a known failure and show CI blocking it with a per-category threshold. Keep the run as proof. Compare CI options in our LLM evaluation tools comparison.
- 13. Online evals: Score a sample of live traffic asynchronously, chart it per failure category, and set one drift alert. See Arize Phoenix vs Langfuse and the Phoenix docs.
- 14. Cost and latency: Log tokens, cost per successful task and p50/p95 latency, swap in a cheaper model, and only claim the saving if quality holds in CI. Langfuse tracks cost.
- 15. Close the loop: Route negative user feedback into a triage queue and turn each confirmed failure into a new labeled case in your dataset, with the date it was added. Compare Braintrust vs LangSmith for review queues.
Tools named in the thread (pick one stack; the concepts carry over): pytest, Pydantic, Promptfoo, DeepEval, Ragas, Inspect, Langfuse, LangSmith, Braintrust, Arize Phoenix.
Reading list: Hamel Husain and Shreya Shankar, AI evals FAQ · Hamel Husain, Your AI product needs evals · Eugene Yan on LLM-evaluators · Eugene Yan on task-specific evals · the τ-bench paper (pass^k).
Where to start this week
Do steps 1, 2 and 4 first: label 50 to 100 real failures, freeze them as version 1 of your dataset and add code checks. Then pick a tool from our LLM evaluation tools comparison that covers your CI gate (step 12).
FAQ
What is an LLM eval checklist?
A list of practices for testing an LLM app: start from real failures, keep a versioned test set, run cheap code checks first, check your LLM judge against human labels, add error bars, gate deploys in CI and keep scoring live traffic.
What is the difference between pass@k and pass^k?
pass@k counts a task as solved if at least one of k attempts succeeds. pass^k counts it only if all k succeed. With an 80% per-attempt success rate and independent attempts, pass^5 is about 33%.
How many test cases do I need for an LLM eval?
Enough that your error bars are smaller than the changes you care about. At 100 cases and 80% accuracy, the 95% interval is about ±8 points, so a 3-point gain is probably noise.
Which tools support gating deploys on eval results?
Promptfoo, DeepEval, LangSmith, Braintrust, Langfuse, Arize and Ragas all document running evals in CI. Links to each vendor's docs are in step 12.
How do I know if my LLM judge is reliable?
Hand-label pass/fail examples, tune the judge on part of them and report its true-positive and true-negative rates on the rest. Langfuse documents comparing judge and human scores with Cohen's kappa and a confusion matrix.
Sources
- diva (@divaagurlxw) on X, 28 Oct 2026 (read 29 Oct 2026)
- diva (@divaagurlxw) on X, follow-up with practice projects, 29 Oct 2026 (read 29 Oct 2026)
- pytest documentation (read 29 Oct 2026)
- Pydantic docs (data validation library for Python) (read 29 Oct 2026)
- Inspect docs (framework for LLM evaluations) (read 29 Oct 2026)
- Arize Phoenix docs: what is Phoenix (read 29 Oct 2026)
- Hamel Husain and Shreya Shankar: AI Evals, everything you need to know (read 29 Oct 2026)
- Hamel Husain: Your AI product needs evals (read 29 Oct 2026)
- Eugene Yan: Evaluating the effectiveness of LLM-evaluators (read 29 Oct 2026)
- Eugene Yan: Task-specific LLM evals that do and don't work (read 29 Oct 2026)
- τ-bench paper on arXiv (2406.12045), which introduces pass^k (read 29 Oct 2026)
- Langfuse docs: datasets (read 29 Oct 2026)
- Langfuse docs: experiments in CI/CD (read 29 Oct 2026)
- Langfuse docs: LLM-as-a-Judge (read 29 Oct 2026)
- Langfuse docs: model usage and cost tracking (read 29 Oct 2026)
- Langfuse pricing (annotation queues) (read 29 Oct 2026)
- LangSmith docs: manage datasets (read 29 Oct 2026)
- LangSmith docs: annotation queues (read 29 Oct 2026)
- LangSmith docs: pytest integration (read 29 Oct 2026)
- LangSmith docs: pairwise evaluation (read 29 Oct 2026)
- Braintrust docs: manage datasets (read 29 Oct 2026)
- Braintrust docs: run in CI (read 29 Oct 2026)
- Braintrust docs: human review (read 29 Oct 2026)
- Opik docs: dataset versioning (read 29 Oct 2026)
- W&B Weave docs: datasets (read 29 Oct 2026)
- Promptfoo docs: deterministic metrics (read 29 Oct 2026)
- Promptfoo pricing (CI/CD in Community) (read 29 Oct 2026)
- DeepEval docs: Golden Synthesizer (read 29 Oct 2026)
- DeepEval docs: Conversation Simulator (read 29 Oct 2026)
- DeepEval docs: Tool Correctness (read 29 Oct 2026)
- DeepEval docs: unit testing in CI/CD (read 29 Oct 2026)
- Ragas docs: available metrics (read 29 Oct 2026)
- Ragas docs: add to CI (read 29 Oct 2026)
- Arize docs: CI/CD for experiments (read 29 Oct 2026)
- Arize pricing (online and offline evals) (read 29 Oct 2026)