DeepEval
by Confident AI
What is DeepEval? DeepEval is an open-source (Apache-2.0) Python framework from Confident AI for testing LLM apps the way pytest tests code. You write test cases, attach metrics such as faithfulness, answer relevancy, G-Eval or task completion, and run them with deepeval test run or in CI.
Editor note
An open-source Python framework for unit-testing LLM apps like pytest, with LLM-judged metrics for RAG (faithfulness, answer relevancy, contextual precision and recall), agents (task completion, tool correctness) and custom criteria (G-Eval).
- What it tests
- Your own LLM app, RAG pipeline or agent, as pass/fail tests in CI.
- Original source
- github.com/confident-ai/deepeval
- Last updated
- Repo last commit 2026-10-15
- License
- Apache-2.0
When to use DeepEval
- You want pass/fail quality checks for an LLM app inside your normal test suite: DeepEval plugs into pytest, and deepeval test run collects eval files the way pytest would.
- You need metrics beyond RAG: the project ships agent metrics (task completion, tool correctness, plan adherence), multi-turn conversation metrics, and custom-criteria metrics (G-Eval, DAG).
- You want RAG metrics with an explanation of each score: answer relevancy, faithfulness, contextual precision, recall and relevancy.
Limitations
- Most metrics use an LLM as the judge, so scores depend on the judge model and prompt; record both.
- Its faithfulness metric counts a claim as truthful if it does not contradict the retrieved context, which is more lenient than Ragas (supported claims only). Scores from the two tools are not interchangeable.
- It is a framework for testing your own system, not a fixed benchmark with a public leaderboard.
- It is built by a company that also sells a hosted platform (Confident AI); the core library works without it.
How to run it
- Install from PyPI.
- Write a test file with LLMTestCase objects and the metrics you want.
- Run it with deepeval test run (or call evaluate() from Python).
pip install -U deepeval
deepeval test run test_app.pyNotable results
- DeepEval's RAGAS metric is the average of four RAG metrics (answer relevancy, faithfulness, contextual precision, contextual recall) and uses the ragas library under the hood. [DeepEval docs: RAGAS metric]
Sources
FAQ
Is DeepEval free?
The library is open source under Apache-2.0. Confident AI, the company behind it, sells a separate hosted platform for reports and collaboration.
Does DeepEval work with pytest?
Yes. It plugs into pytest; deepeval test run collects and runs eval files the same way pytest would, so it fits into CI.
What metrics does DeepEval have?
Custom-criteria metrics (G-Eval, DAG), RAG metrics (answer relevancy, faithfulness, contextual precision, recall, relevancy), agent metrics (task completion, tool correctness, plan adherence and more) and multi-turn conversation metrics.
Compare DeepEval
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related rubric libraries & frameworks evals
- Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
- OpenAI Evals — Building and running your own evals; community registry.
- LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
- Prompt Assay judge templates — Starter LLM-as-judge prompts.