How to choose an eval for your use case

A public benchmark score tells you how a model does on someone else's problem. This guide helps you pick evals that tell you whether your product works.

Quick picker: what are you building?

Start with: τ-bench, LLMJury rubrics, IFEval

Write your own policy rules into test cases; generic benchmarks won't know your refund policy.

1. Define the task and how it fails

Write one sentence describing what your AI must do, for example “answer refund questions following our policy”. Then list 5–10 concrete ways it could fail: wrong facts, ignoring policy, unsafe advice, wrong language, refusing a valid request, broken format. Look at real outputs or logs if you have them. These failure modes are what your eval must catch.

2. Pick the matching category

Match your task to a category: coding, agents, customer support, writing, safety, healthcare, multilingual or RAG. Public benchmarks are good for comparing models before you build. Your own test set matters more once you have a product.

3. Check freshness and saturation

When top models all score near the maximum, a benchmark stops telling them apart. It is “saturated”. Older public test sets may also have leaked into training data (“contamination”). Prefer evals that are actively maintained or add new items over time, such as LiveCodeBench, or that were built to be hard, such as Humanity's Last Exam. Each listing shows the last update date where we have checked it.

4. Check the license

Licenses vary. Some allow commercial use, some don't, and some datasets ask you not to publish examples online. Read the license at the original source before you use an eval in a product or share its data. We show the license where we have verified it.

5. Estimate the cost to run

Cost comes from three places: model calls (number of test items × repeats × tokens), judge calls if a model grades the answers, and environment setup for agent evals such as WebArena or OSWorld, which need hosted websites or virtual machines. Start with a small sample, perhaps 50–100 items, to see the cost before running everything.

6. Decide: automatic checks, LLM judge or humans

  • Automatic checks (unit tests, exact answers, format rules) are cheap and consistent. Use them whenever a right answer can be checked by code, like SWE-bench or IFEval.
  • LLM-as-judge scales to open-ended answers but has known biases, such as favouring longer answers. See the MT-Bench paper. Calibrate it against a sample of human ratings.
  • Human experts are slow and costly but essential for high-stakes areas like health, and for languages where judges are weaker. SAKHI found doctors and LLM judges disagreeing on individual answers.

A common pattern is to automate everything you can, use an LLM judge for the rest, and have humans review borderline and high-risk cases.

7. Build your own rubric

  1. Turn each failure mode from step 1 into one yes/no criterion, for example “Tells the user to seek emergency care when symptoms are red flags”.
  2. Give each criterion a weight. Make serious failures count heavily, or fail the whole answer.
  3. Write two or three example answers that pass and that fail.
  4. Collect 30–100 real or realistic test cases, including hard and edge cases.
  5. Score a sample by hand, then check your LLM judge agrees with you before trusting it at scale.
  6. Re-run after every model or prompt change, and add new cases when you find new failures.

Good starting points: HealthBench (a well-documented rubric design), LLMJury rubrics, Prompt Assay templates, and frameworks like Inspect Evals, OpenAI Evals and Ragas.

Quick checklist

  • Does it test my real failure modes?
  • Is it still hard for current models, and fresh?
  • Does the license allow my use?
  • Can I afford to run it regularly?
  • Is the grading method trustworthy for my stakes?
  • Do I have my own small test set too?