Promptfoo

by Promptfoo (now part of OpenAI)

What is Promptfoo? Promptfoo is an open-source (MIT) command-line tool and library for evaluating and red-teaming LLM apps. You declare prompts, providers and test cases with assertions in a YAML config, run promptfoo eval, and compare outputs across models in a matrix view or in CI. Promptfoo is now part of OpenAI and remains open source.

Page last updated 2026-10-16 · Evals Review

Editor note

An open-source (MIT) command-line tool for testing prompts, agents and RAG apps from declarative YAML configs, comparing models side by side, running checks in CI, and red-teaming LLM apps for vulnerabilities.

What it tests: Your own prompts, models and LLM apps, with deterministic and LLM-graded assertions.
Original source: github.com/promptfoo/promptfoo
Last updated: Repo last commit 2026-10-16
License: MIT

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators.

When to use Promptfoo

You want to compare prompts or models side by side (OpenAI, Anthropic, Azure, Bedrock, Ollama and more) from one config file.
You want assertions without writing Python: deterministic checks plus LLM-graded ones such as llm-rubric, g-eval, factuality and context-faithfulness.
You need security testing: Promptfoo's red-teaming mode scans LLM apps for vulnerabilities.

Limitations

LLM-graded assertions (llm-rubric, model-graded-*) need a grader model, and carry its biases and cost.
It is a test harness, not a benchmark: scores depend entirely on the test cases you write.

How to run it

Install with npm, brew or pip, or use npx.
Create a config with promptfoo init, add prompts, providers and tests with assertions.
Run promptfoo eval, then promptfoo view to compare results.

npx promptfoo@latest init
npx promptfoo@latest eval
npx promptfoo@latest view

Notable results

README (read 16 Oct 2026): Promptfoo is now part of OpenAI and remains open source and MIT licensed. [Official repository]

Sources

Everything above comes from the sources listed, read on 15–16 Oct 2026. Commands are copied from the official instructions; check the source for changes.

FAQ

Is Promptfoo free?

The CLI and library are open source under the MIT license.

Is Promptfoo owned by OpenAI?

Promptfoo's README says it is now part of OpenAI and remains open source and MIT licensed.

Does Promptfoo support LLM-as-a-judge?

Yes. Assertions such as llm-rubric, g-eval, factuality and model-graded-closedqa use a grader model.

Compare Promptfoo

DeepEval vs Promptfoo

Reviews

No reviews yet.

Reviews from practitioners who sign in with GitHub open soon.

Related rubric libraries & frameworks evals

Inspect Evals — Many benchmarks under one runner: coding, agents, safety, knowledge.
OpenAI Evals — Building and running your own evals; community registry.
LLMJury rubric library — LLM-as-judge rubrics for support, RAG, summaries, code and tone.
Prompt Assay judge templates — Starter LLM-as-judge prompts.

All Rubric libraries & frameworks evals · Comparison table