How to write AI evals: eval criteria and rubrics for product managers

Short answer: write AI evals from real failures, not from a list of virtues: turn each failure into one pass/fail question a reviewer can answer from the output alone, add a passing and a failing example, choose the cheapest grader that can catch it, and fix the launch bar before you see results. Updated 11 Nov 2026.

By Evals Review editorial team · Published 11 Nov 2026 · Updated 11 Nov 2026

How we checked: tool features link to vendor docs or pricing pages read on 11 Nov 2026; advice on judge scales cites LangChain's eval-driven development guide. The rubric template is our own illustrative example, not data from a real product. How we review.

Seven steps to an eval a PM can write and an engineer can run

  1. Start from real failures, not from a list of qualities. Read a sample of real outputs and write one plain note on each bad one. Group the notes and count them. "Helpful" and "accurate" are too broad to grade; "quotes a refund window that isn't in our policy" is gradeable. The method is step 1 of our eval checklist.
  2. Write one pass/fail criterion per failure. Each criterion is a yes/no question a reviewer can answer from the output alone. Prefer pass/fail to a 1-10 scale: LangChain's guide recommends narrow pass/fail questions over an ambiguous 1 to 10 score for LLM judges. [LangChain blog] If you need more than two levels, use named categories (correct, partly correct, wrong) rather than numbers; Langfuse, for one, supports boolean and categorical judge scores. [Langfuse docs]
  3. Add the when, the do, the don't and the must-keep. For agent behaviors, a four-line spec makes the criterion testable: when the behavior applies, what the agent should do, what it must not do, and which current behaviors must keep working. [LangChain blog] The "must keep working" line is what stops a fix from quietly breaking something else.
  4. Write a passing and a failing example for each criterion. Two or three short examples per side are enough to show reviewers and judges where the line is. Use real outputs where you can, with personal data removed.
  5. Pick the cheapest grader that can see the failure. Code checks for anything deterministic (format, required fields, the right tool called, banned phrases). An LLM judge for judgement calls such as tone or whether an answer is supported by the source. People for high-stakes or ambiguous cases, and to label the examples you calibrate the judge on.
  6. Check the judge against your own labels. Label a set of outputs yourself, run the judge on the same set and read every disagreement. Tighten the criterion wording until disagreements are rare, then keep a few labelled cases to re-check whenever the judge model or prompt changes. [LangChain blog]
  7. Set the launch bar before you see results. Decide in writing which criteria are blockers (any failure stops a release) and what pass rate the rest need. Setting it after the numbers arrive turns the bar into a negotiation.

Rubric template (worked example)

An illustrative rubric for a customer-support assistant. Copy the columns; replace the rows with your own failures.

Example eval rubric: one pass/fail criterion per row
Failure it catchesPass/fail questionPassesFailsGrader and weight
Quotes only policy that existsEvery refund or warranty term in the answer appears in the policy docStates the 30-day window from the docInvents a 60-day windowLLM judge with the policy as context; blocker
Escalates when it shouldHands off to a person when the user asks for one or reports a safety issue"I'll connect you with an agent now"Keeps troubleshooting after "let me talk to a human"Code check on the handoff tool call
Doesn't ask for data it shouldn'tNever asks for a full card number or passwordAsks for the order numberAsks for the card's CVVCode check (pattern match)
Answers the actual questionThe first sentence addresses what the user askedLeads with the delivery date asked forOpens with a generic greeting and a policy summaryLLM judge; target pass rate set before launch

Where PMs can write criteria without code

Price and seat comparison: LLM evaluation tools for product managers. Rubric libraries to borrow structure from: LLMJury rubrics, Prompt Assay judge templates, HealthBench. More in rubric libraries and frameworks.

Next: run the criteria in a loop

Criteria only pay off when every change is scored against them. That loop is eval-driven development: write the eval first, run a baseline, change one thing, compare example by example.

Evals for AI product teams: the full series

FAQ

How do I write an eval as a product manager?

Start from real failures in your product's outputs, turn each into one pass/fail question a reviewer can answer from the output alone, add a passing and a failing example, pick a grader (code, LLM judge or person), check the judge against your own labels, and set the launch bar before you see results.

Should AI eval rubrics use pass/fail or a 1-10 scale?

Pass/fail by default. Two reviewers agree more easily on yes or no than on a 6 versus a 7, and LangChain's eval-driven development guide recommends narrow pass/fail questions over a 1 to 10 scale for LLM judges. Use named categories when you need more than two levels.

How many criteria should an AI eval have?

As many as you have distinct failure types you care about, and no more. Each criterion should trace back to a failure you saw or a behavior you are building; a criterion nobody can link to a real failure is a candidate to drop.

Who should write eval criteria, PM or engineer?

Usually the PM or domain expert writes the criteria and the launch bar, because they decide what good means for users. Engineers turn the criteria into code checks and judge prompts and wire them into CI.

Where can I find example rubrics?

Public rubric libraries such as the LLMJury rubric library and Prompt Assay's judge templates are listed on Evals Review, and HealthBench shows a large physician-written rubric design. Adapt their structure, not their wording, to your own failures.

Sources