Quickstart reality check: do eval tools' official quickstarts work as written?
Short answer: on 11 Nov 2026, with a free local model instead of a paid API, only Promptfoo's official quickstart reached a first eval result with no extra fix: 3 steps, 27 seconds. DeepEval, Ragas, Inspect AI and lm-evaluation-harness all got there too, but each needed 2 or 3 fixes their docs don't mention. Ragas's recommended first command crashed on a dependency, which would also stop someone using an OpenAI key.
The question, and why it matters
When you shortlist eval tools, the quickstart is usually your first hour with each one. If it breaks, you lose time or drop the tool before seeing what it can do. Most quickstarts assume an OpenAI key. Many teams can't send data to a hosted model, or want to try before paying, so we also checked how each one handles a free local model.
Results per tool
| Tool (version that ran) | Steps run | Failed steps | Fixes not in the docs | Setup time | Eval time | Time to first result | First result printed |
|---|---|---|---|---|---|---|---|
| Promptfoo (0.125.1) | 3 | 0 | 0 | 23.0 s | 3.7 s | 26.7 s | 2 of 2 tests passed |
| DeepEval (4.3.3) | 10 | 3 | 2 | 35.1 s | 11.9 s | 47.1 s | 0 of 1 test passed (the run finished; the judge scored the sample below the 0.5 threshold) |
| Ragas (0.4.3) | 9 | 3 | 3 | 1 min 40 s | 28.8 s | 2 min 9 s | 0 of 3 sample questions scored pass |
| Inspect AI (0.3.289) | 7 | 1 | 2 | 52.7 s | 6 min 36 s | 7 min 29 s | accuracy 0.30 on the first 10 SimpleQA questions |
| lm-evaluation-harness (0.4.14.dev0 (commit d6de8164)) | 7 | 2 | 3 | 1 min 4 s | 1 min 9 s | 2 min 12 s | gsm8k exact match 0.4 on the first 5 questions |
How to read it: "Steps run" counts every command, including retries after a failure. Setup time runs from the first install command to the start of the successful eval. Eval time is mostly the local model working on a CPU: Inspect's 6.6 minutes are 10 questions plus 10 grading calls, one at a time. The "first result" scores come from a 3B model on sample data. They show the tool ran, not how good the tool is.
What broke, and the fix that worked
The last column says whether the problem would also hit someone following the quickstart with a paid OpenAI key. "Setup" means it depends on the machine: system Python on recent Debian and Ubuntu refuses pip installs outside a virtual environment.
| Tool | What happened | Fix | Hits key users too? |
|---|---|---|---|
| DeepEval | The quickstart says to run cp .env.example .env.local. A new project has no .env.example, so the command fails. | None needed for a local run; we carried on. | yes |
| DeepEval | deepeval set-ollama --model=qwen2.5:3b, as the linked judge docs show, reported success. The test still failed with "LocalModel is missing a required parameter". | Run it again with --save=dotenv, so the setting that marks the judge as Ollama is written to .env.local. | no |
| DeepEval | Next run: "OllamaModel requires the ollama package". The quickstart and the judge docs don't mention it. | pip install ollama | no |
| Ragas | The recommended first command, uvx ragas quickstart rag_eval, crashed on import: No module named 'langchain_community.chat_models.vertexai'. Ragas 0.4.3 pulls langchain-community 0.4.2, which no longer has that module. | uvx --with "langchain-community<0.4" ragas quickstart rag_eval | yes |
| Ragas | Same crash inside the generated project after uv sync. | uv add "langchain-community<0.4" (0.3.31 installed) | yes |
| Ragas | The Ollama tab only swaps the judge. The sample app still builds an OpenAI client and calls gpt-4o, so it stops on "Missing credentials". | Point the sample app's client and model at Ollama too (two lines in evals.py and rag.py). | no |
| Inspect AI | pip install inspect-ai is refused by system Python on Debian/Ubuntu (externally managed environment) unless you are in a virtual environment. The docs don't create one. | python3 -m venv .venv first | setup |
| Inspect AI | The SimpleQA example stopped with "Hugging Face Datasets requires optional dependencies". | pip install datasets | yes |
| lm-evaluation-harness | No virtual environment step before pip install -e . (same Debian/Ubuntu issue). | python3 -m venv .venv first | setup |
| lm-evaluation-harness | The README's local-completions line puts batch_size=16 inside --model_args. This version rejects it: "got multiple values for keyword argument 'batch_size'". | Drop batch_size from --model_args | yes |
| lm-evaluation-harness | Next: No module named 'transformers'. local-completions loads a Hugging Face tokenizer named after the model, and an Ollama tag isn't one. | Add tokenizer_backend=None to --model_args | no |
Two smaller notes. Ragas prints a results path (experiments/) that is not where the file lands (evals/experiments/). Promptfoo exits with code 100, and DeepEval with code 1, when a test fails, so in CI the job fails even though the eval ran.
How we pointed each quickstart at a local model
Every quickstart defaults to a paid hosted model. We used each vendor's own documented local route and changed nothing else:
- Promptfoo: Replaced the two openai: providers in promptfooconfig.yaml with ollama:chat:qwen2.5:3b (Promptfoo's own Ollama provider).
- DeepEval: deepeval set-ollama --model=qwen2.5:3b, from DeepEval's "Configure LLM judges" docs, instead of exporting OPENAI_API_KEY.
- Ragas: The quickstart's own "Local Models (Ollama)" snippet: an OpenAI-compatible client at localhost:11434/v1 passed to llm_factory.
- Inspect AI: --model ollama/qwen2.5:3b instead of openai/…, plus --limit 10, so the 1,000-question SimpleQA set finishes in minutes on a CPU.
- lm-evaluation-harness: The README's own local-server route (lm_eval[api] + local-completions) at Ollama's /v1/completions, on gsm8k with --limit 5. The README's first example needs a CUDA GPU (--device cuda:0); this machine has none.
What it means if you're picking a tool
- Expect to debug the quickstart, not just run it. Four of five needed changes their docs don't mention. Allow about an hour per tool, not the "5 minutes" some pages promise.
- Local-model support exists everywhere, but it's the least tested path. Half of the fixes (5 of 10) only matter once you leave the OpenAI default.
- Pin your versions. The Ragas crash came from a new version of a dependency, not from Ragas itself. A pinned lockfile would have avoided it.
- The quickstart is not the product. A rough first hour says nothing about metric quality, tracing or how a tool handles real datasets. For those, see Ragas vs DeepEval, DeepEval vs Promptfoo and the eval tools for product managers roundup. To start writing your own evals, see eval-driven development.
Method
| Date and time | 11 Nov 2026, 07:01–07:15 IST, one tool after another |
|---|---|
| Machine | Debian 13 (Linux 6.12), Intel Xeon, 8 vCPU, 15 GiB RAM, no GPU |
| Model | Ollama 0.40.16, qwen2.5:3b (id 357c53fb659c, Q4_K_M), loaded before each tool |
| Runtimes | Python 3.13.5 (the uv-managed Ragas project used 3.12), Node 24.22.1, npm 11.21.1, uv 0.13.7 |
| Tool versions | Whatever each quickstart installed that morning: Promptfoo 0.125.1, DeepEval 4.3.3, Ragas 0.4.3, Inspect AI 0.3.289, lm-evaluation-harness 0.4.14.dev0 (commit d6de8164) |
| Isolation | A fresh empty folder per tool, empty npm/pip/uv/Hugging Face caches and a new virtual environment per Python tool. API keys removed from the environment. |
- Saved the text of each official quickstart as it read on 11 Nov 2026.
- Typed each command as written. Our changes are logged separately: adapt for the local model, fix for anything the docs miss.
- When a step failed, we kept the failure in the count, applied the smallest fix we found, and continued.
- Stopped the clock when the tool printed its own end-of-run summary. A run with a failing test still counts as a result.
- Recorded every terminal session with asciinema.
Limits
- One timed run per tool on one machine. Seconds will differ on yours, and our box runs other jobs too. The step counts and breaks are the solid part.
- The quickstarts don't fix temperature, so their pass/fail numbers move between runs. In earlier untimed runs, Promptfoo scored 1 of 2 and Inspect 0.0.
- "Latest" changes. A quickstart that broke on 11 Nov 2026 may be fixed next week, or a working one may break.
- A CPU and a 3B model make eval time long and scores low. With a hosted model, eval time would be seconds. Setup time would barely change.
- Only the main getting-started path was used: Promptfoo's example project, DeepEval's single-turn LLM-as-a-judge tab, Ragas's uvx route, Inspect's SimpleQA example, and lm-evaluation-harness's API route rather than its GPU example.
Rerun it yourself
Everything is in one folder: a script per tool, the timing harness, raw results.csv, full logs, terminal recordings and the quickstart text we followed.
ollama pull qwen2.5:3b
tar xzf quickstart-reality-check.tar.gz && cd quickstart-reality-check
./run-all.sh # all five, recorded; writes results.csv
./run-ragas.sh # or one toolFAQ
Which eval tool quickstart is easiest to run?
In our 11 Nov 2026 run, Promptfoo was the only one of five that reached a first result with no fix the docs don't mention: 3 steps and 27 seconds with a local model. That is one run on one machine, and it says nothing about which tool evaluates better.
Can you run these eval tools without an OpenAI API key?
Yes, all five reached a result with a free local model served by Ollama. Each needed its own local-model setting, and four of the five needed at least one extra fix we list on this page.
Does this test say which eval tool is best?
No. It measures only the official getting-started path: steps, breaks and time to the first result. The scores the tools printed come from a 3-billion-parameter local model on sample data, not from the tools.
How do I rerun this test?
Download the scripts, start Ollama with qwen2.5:3b, and run ./run-all.sh. It creates a fresh folder and cold caches per tool, logs every step with timings and writes results.csv.