Rubric libraries & frameworks

Pydantic Evals

by Pydantic (pydantic-evals package in the pydantic/pydantic-ai repository)

What is Pydantic Evals? Pydantic Evals is a rubric libraries & frameworks eval made by Pydantic (pydantic-evals package in the pydantic/pydantic-ai repository). A code-first Python framework for testing LLM apps and agents. You define datasets, cases, tasks and evaluators in Python (or load them from files), run an experiment, and read the report in your terminal or, optionally, in Pydantic Logfire. Built-in evaluators cover checks such as exact match and output type, next to LLM-as-a-judge and custom evaluators. Span-based evaluators read OpenTelemetry traces, so you can grade how an agent reached its answer, such as the order and arguments of its tool calls, and not only the final output. Online evaluation attaches evaluators to all or a sample of live or staging calls. The pydantic-evals package installs on its own and does not require Pydantic AI.

Page last updated 2026-10-18 · Evals Review

Editor note

A code-first Python framework for testing LLM apps and agents. You define datasets, cases, tasks and evaluators in Python (or load them from files), run an experiment, and read the report in your terminal or, optionally, in Pydantic Logfire. Built-in evaluators cover checks such as exact match and output type, next to LLM-as-a-judge and custom evaluators. Span-based evaluators read OpenTelemetry traces, so you can grade how an agent reached its answer, such as the order and arguments of its tool calls, and not only the final output. Online evaluation attaches evaluators to all or a sample of live or staging calls. The pydantic-evals package installs on its own and does not require Pydantic AI.

What it tests
Whatever you define: output checks, LLM-judged quality and agent tool-call trajectories, offline on datasets or online on sampled traffic.
Original source
pydantic.dev/docs/ai/evals/evals/
Last updated
pydantic-evals 2.71.0 on PyPI (read 19 Nov 2026); repo pydantic/pydantic-ai last push 2026-11-19 (GitHub API, read 19 Nov 2026)
License
MIT (pydantic/pydantic-ai, GitHub API, read 19 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related rubric libraries & frameworks evals

All Rubric libraries & frameworks evals · Comparison table

Discussion

No one has discussed Pydantic Evals yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions