Promptfoo vs DeepEval / Confident AI
Prices and features checked 2026-10-17 on both vendors' own pages. “not verified” means we could not confirm it.
Promptfoo or DeepEval: pricing and platform? Both start as free open-source tools: Promptfoo is an MIT command-line tool configured in YAML, with red-teaming up to 10k probes a month free; DeepEval is an Apache-2.0 Python framework with pytest-style tests. The paid layers differ: Promptfoo Enterprise is custom-priced, while DeepEval's hosted platform, Confident AI, publishes a free plan and Starter at $200/month per organization.
| Promptfoo | DeepEval / Confident AI | |
|---|---|---|
| Made by | Promptfoo (now part of OpenAI) | Confident AI |
| Licence | MIT [source] | DeepEval: Apache-2.0. Confident AI: hosted service [source] |
| Self-hosting | Run locally or self-host; on-premise Enterprise tier [source] | DeepEval runs locally; Confident AI on-prem on Enterprise [source] |
| Meter | Red-team probes (10k/month free); Enterprise priced on team size and needs [source] | Flat monthly fee per organization plus trace spans in GB-months [source] |
| Free tier | Community: free, all LLM evaluation features, red teaming up to 10k probes/month [source, checked 2026-10-17] | Free: 2 seats, 1 project, 5 test runs/week, 1 GB-month of trace spans [source, checked 2026-10-17] |
| Entry paid price | Enterprise: custom pricing (no public price) [source, checked 2026-10-17] | Starter $200/month per organization: unlimited seats, 5 projects, 5 GB-months, then $1 per GB-month [source, checked 2026-10-17] |
| Retention | not verified | Retention billed in GB-months; adjustable [source] |
| OpenTelemetry | OpenTelemetry tracing in evaluations [source] | Confident AI docs include an Integrations & OTEL section [source] |
| CI gating | CI/CD integration in the free Community version [source] | Unit testing in CI/CD [source] |
| LLM-as-judge | Model-graded assertions (rubrics, faithfulness, relevance) [source] | G-Eval and LLM-judged RAG and agent metrics [source] |
| Human annotation | not verified | Annotation queues on Starter and above [source] |
| Prompt management | not verified | Prompt versioning on the free plan [source] |
| Dataset versioning | not verified | Metric and dataset versioning on Team ($2,000/month) [source] |
| SOC 2 / HIPAA | not verified | SOC 2 on Team; HIPAA on Enterprise [source] |
Verdict: when to use which
Use Promptfoo if you want config-driven tests across many models and providers plus red-teaming, mostly run locally or in CI.
Use DeepEval / Confident AI if you write tests in Python and want a hosted dashboard with published prices for test reports, tracing, annotation and prompt versioning.
The meters differ, so price both on a week of your own traffic in the pricing calculator.
FAQ
Which has a hosted plan with a public price?
Confident AI (for DeepEval): Free, then Starter at $200/month and Team at $2,000/month. Promptfoo lists Enterprise and On-Premise as custom pricing.
Can both run in CI?
Yes. Promptfoo lists CI/CD integration in its free Community version; DeepEval documents unit testing in CI/CD.
Where do I compare how tests are written?
See our DeepEval vs Promptfoo page, which covers metrics, assertions and test syntax.
More
Other tool comparisons: Braintrust vs Langfuse · Langfuse vs LangSmith · Braintrust vs LangSmith · Arize Phoenix vs Langfuse · Comet Opik vs Langfuse