vitest-evals
by Sentry (getsentry)
What is vitest-evals? vitest-evals is a rubric libraries & frameworks eval made by Sentry (getsentry). A Vitest extension from Sentry for running evals of TypeScript AI apps and agents as ordinary test suites. Each suite is bound to one harness (first-party adapters cover the AI SDK, OpenAI Agents and pi-ai), runs your app on an input and returns a normalised transcript with tool calls and trace spans that you assert on directly. Optional judges, including LLM-backed ones with a score threshold, run after each case. A reporter, a local report viewer and a GitHub Action turn the Vitest JSON output into job summaries, annotations for failed evals and an optional Check Run with a minimum pass rate.
Editor note
A Vitest extension from Sentry for running evals of TypeScript AI apps and agents as ordinary test suites. Each suite is bound to one harness (first-party adapters cover the AI SDK, OpenAI Agents and pi-ai), runs your app on an input and returns a normalised transcript with tool calls and trace spans that you assert on directly. Optional judges, including LLM-backed ones with a score threshold, run after each case. A reporter, a local report viewer and a GitHub Action turn the Vitest JSON output into job summaries, annotations for failed evals and an optional Check Run with a minimum pass rate.
- What it tests
- Agent and LLM-app behaviour in a TypeScript codebase: outputs, tool calls and judge scores, gated in CI.
- Original source
- github.com/getsentry/vitest-evals
- Last updated
- npm vitest-evals 0.18.1 (registry, read 19 Nov 2026); repo last push 2026-11-07 (GitHub API, read 19 Nov 2026)
- License
- Apache-2.0 (GitHub API, read 19 Nov 2026)
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.
Discussion
No one has discussed vitest-evals yet. Have you run it? Share what it caught, what it missed or how you set it up.