DeepEval vs Promptfoo
DeepEval or Promptfoo? Both are open-source tools for testing your own LLM app. DeepEval (Python, Apache-2.0) is pytest-style: you write test cases in code with metrics such as faithfulness or G-Eval. Promptfoo (MIT, CLI with Node, brew or pip installs) is config-first: you declare prompts, providers and assertions in YAML, compare models in a matrix, and can red-team the app. Pick DeepEval for Python test suites, Promptfoo for config-driven model comparison and security scans.
| DeepEval | Promptfoo | |
|---|---|---|
| Made by | Confident AI | Promptfoo (now part of OpenAI) |
| Maintainer | Confident AI | Promptfoo, now part of OpenAI |
| License | Apache-2.0 | MIT |
| How you write tests | Python test cases, pytest-style | Declarative YAML config |
| Run | deepeval test run | promptfoo eval, promptfoo view |
| LLM-judged metrics | Faithfulness, answer relevancy, contextual precision/recall, G-Eval, DAG, task completion, tool correctness | llm-rubric, g-eval, factuality, context-faithfulness, context-recall, model-graded-closedqa |
| Red teaming | Not compared here | Built-in red-teaming and vulnerability scanning |
| CI | Runs inside pytest | CLI and CI/CD integration |
| License | Apache-2.0 | MIT |
Verdict: when to use which
Use DeepEval if your team writes Python and wants LLM checks inside the existing test suite.
Use Promptfoo if you want to compare prompts and models from one config, or scan an app for vulnerabilities.
Dated numbers
- Promptfoo README (read 16 Oct 2026): now part of OpenAI, still open source and MIT licensed. [Promptfoo repository]
- Promptfoo docs: llm-rubric and model-graded-* assertions require an LLM grader provider. [Promptfoo docs: assertions]
- DeepEval docs (read 16 Oct 2026): faithfulness = truthful claims ÷ total claims against the retrieved context. [DeepEval docs: faithfulness]
FAQ
Can I use DeepEval and Promptfoo together?
Yes. They don't conflict; some teams use DeepEval for unit-style tests and Promptfoo for model comparison and red teaming. Keep their scores separate, since metric definitions differ.
Do both need an LLM to grade?
Their LLM-judged metrics do. Promptfoo also has deterministic assertions that don't need a grader model.
Sources
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · WebArena vs OSWorld · HarmBench vs StrongREJECT · GAIA vs BrowseComp · SWE-Bench Pro vs SWE-bench Verified · Terminal-Bench vs SWE-bench · MMLU vs MMLU-Pro · GPQA vs Humanity's Last Exam · Ragas vs DeepEval · Terminal-Bench 2.1 vs Terminal-Bench 4.0 · ARC-AGI-2 vs ARC-AGI-3 · LiveCodeBench vs SWE-bench · LLM-as-a-Judge vs Agent-as-a-Judge