LLM-as-judge tools, rubrics and eval frameworks
These aren't benchmarks with fixed leaderboards: they are frameworks, rubric collections and judge tools you use to build and score your own evals. Vendor-made entries are labelled in the summaries.
| Eval | Made by | What it tests | License | Last updated | Source |
|---|---|---|---|---|---|
| Inspect Evals | UK AI Security Institute & community | Many benchmarks under one runner: coding, agents, safety, knowledge. | MIT | Repo last commit 2026-10-15 | github.com |
| OpenAI Evals | OpenAI | Building and running your own evals; community registry. | — | Repo last commit 2026-04-14 | github.com |
| LLMJury rubric library | LLMJury | LLM-as-judge rubrics for support, RAG, summaries, code and tone. | — | — | llmjury.com |
| Prompt Assay judge templates | Prompt Assay | Starter LLM-as-judge prompts. | — | — | promptassay.ai |
| Jev (TypeSafe AI) | TypeSafe AI | Not a benchmark: a tool for AI-judge / LLM-as-judge style scoring in your own evals. | — | Jev 1.14.0 announced 2026-10-14 | docs.typesafe.ai |
In short
Inspect Evals — A large, actively maintained collection of ready-to-run evals implemented in the Inspect framework.
OpenAI Evals — A framework and open registry of evals, including templates for model-graded evaluation.
LLMJury rubric library — Paste-ready judge rubrics for questions like 'did the support reply resolve the request?', each with score meanings, examples and known failure modes. Made by a vendor.
Prompt Assay judge templates — Copy-paste YAML judge templates covering basics like relevance and tone plus RAG, code, summarisation and agent traces. Made by a vendor.
Jev (TypeSafe AI) — A commercial model API built for structured judgments rather than text: you send a state plus typed questions (pick a choice, score against a rubric, true/false likelihood) and get back typed answers with probabilities and confidence. Version 1.14.0 (announced 14 Oct 2026) adds experimental image inputs, a bounding-box answer type and longer context behind opt-in flags. Made by a vendor; paid API.
Browse the full Rubric libraries & frameworks category, or see other comparisons.