Rubric libraries & frameworks

vitest-evals

by Sentry (getsentry)

What is vitest-evals? vitest-evals is a rubric libraries & frameworks eval made by Sentry (getsentry). A Vitest extension from Sentry for running evals of TypeScript AI apps and agents as ordinary test suites. Each suite is bound to one harness (first-party adapters cover the AI SDK, OpenAI Agents and pi-ai), runs your app on an input and returns a normalised transcript with tool calls and trace spans that you assert on directly. Optional judges, including LLM-backed ones with a score threshold, run after each case. A reporter, a local report viewer and a GitHub Action turn the Vitest JSON output into job summaries, annotations for failed evals and an optional Check Run with a minimum pass rate.

Page last updated 2026-10-18 · Evals Review

Editor note

A Vitest extension from Sentry for running evals of TypeScript AI apps and agents as ordinary test suites. Each suite is bound to one harness (first-party adapters cover the AI SDK, OpenAI Agents and pi-ai), runs your app on an input and returns a normalised transcript with tool calls and trace spans that you assert on directly. Optional judges, including LLM-backed ones with a score threshold, run after each case. A reporter, a local report viewer and a GitHub Action turn the Vitest JSON output into job summaries, annotations for failed evals and an optional Check Run with a minimum pass rate.

What it tests
Agent and LLM-app behaviour in a TypeScript codebase: outputs, tool calls and judge scores, gated in CI.
Original source
github.com/getsentry/vitest-evals
Last updated
npm vitest-evals 0.18.1 (registry, read 19 Nov 2026); repo last push 2026-11-07 (GitHub API, read 19 Nov 2026)
License
Apache-2.0 (GitHub API, read 19 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related rubric libraries & frameworks evals

All Rubric libraries & frameworks evals · Comparison table

Discussion

No one has discussed vitest-evals yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions