Pydantic Evals
by Pydantic (pydantic-evals package in the pydantic/pydantic-ai repository)
What is Pydantic Evals? Pydantic Evals is a rubric libraries & frameworks eval made by Pydantic (pydantic-evals package in the pydantic/pydantic-ai repository). A code-first Python framework for testing LLM apps and agents. You define datasets, cases, tasks and evaluators in Python (or load them from files), run an experiment, and read the report in your terminal or, optionally, in Pydantic Logfire. Built-in evaluators cover checks such as exact match and output type, next to LLM-as-a-judge and custom evaluators. Span-based evaluators read OpenTelemetry traces, so you can grade how an agent reached its answer, such as the order and arguments of its tool calls, and not only the final output. Online evaluation attaches evaluators to all or a sample of live or staging calls. The pydantic-evals package installs on its own and does not require Pydantic AI.
Editor note
A code-first Python framework for testing LLM apps and agents. You define datasets, cases, tasks and evaluators in Python (or load them from files), run an experiment, and read the report in your terminal or, optionally, in Pydantic Logfire. Built-in evaluators cover checks such as exact match and output type, next to LLM-as-a-judge and custom evaluators. Span-based evaluators read OpenTelemetry traces, so you can grade how an agent reached its answer, such as the order and arguments of its tool calls, and not only the final output. Online evaluation attaches evaluators to all or a sample of live or staging calls. The pydantic-evals package installs on its own and does not require Pydantic AI.
- What it tests
- Whatever you define: output checks, LLM-judged quality and agent tool-call trajectories, offline on datasets or online on sampled traffic.
- Original source
- pydantic.dev/docs/ai/evals/evals/
- Last updated
- pydantic-evals 2.71.0 on PyPI (read 19 Nov 2026); repo pydantic/pydantic-ai last push 2026-11-19 (GitHub API, read 19 Nov 2026)
- License
- MIT (pydantic/pydantic-ai, GitHub API, read 19 Nov 2026)
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.
Discussion
No one has discussed Pydantic Evals yet. Have you run it? Share what it caught, what it missed or how you set it up.