Quickstart reality check: do eval tools' official quickstarts work as written?

Short answer: on 11 Nov 2026, with a free local model instead of a paid API, only Promptfoo's official quickstart reached a first eval result with no extra fix: 3 steps, 27 seconds. DeepEval, Ragas, Inspect AI and lm-evaluation-harness all got there too, but each needed 2 or 3 fixes their docs don't mention. Ragas's recommended first command crashed on a dependency, which would also stop someone using an OpenAI key.

By Evals Review editorial team · Run 11 Nov 2026, 07:01–07:15 IST · Published 11 Nov 2026

What this is: a hands-on run of each vendor's getting-started page on one machine, timed and logged. What it isn't: a verdict on tool quality. No affiliate links on this page (disclosure).

The question, and why it matters

When you shortlist eval tools, the quickstart is usually your first hour with each one. If it breaks, you lose time or drop the tool before seeing what it can do. Most quickstarts assume an OpenAI key. Many teams can't send data to a hosted model, or want to try before paying, so we also checked how each one handles a free local model.

Watch the 60-second video of this test (AI-voiced narration)

Results per tool

Official quickstarts, run 11 Nov 2026 with Ollama qwen2.5:3b on an 8 vCPU machine with no GPU
Tool (version that ran)Steps runFailed stepsFixes not in the docsSetup timeEval timeTime to first resultFirst result printed
Promptfoo (0.125.1)30023.0 s3.7 s26.7 s2 of 2 tests passed
DeepEval (4.3.3)103235.1 s11.9 s47.1 s0 of 1 test passed (the run finished; the judge scored the sample below the 0.5 threshold)
Ragas (0.4.3)9331 min 40 s28.8 s2 min 9 s0 of 3 sample questions scored pass
Inspect AI (0.3.289)71252.7 s6 min 36 s7 min 29 saccuracy 0.30 on the first 10 SimpleQA questions
lm-evaluation-harness (0.4.14.dev0 (commit d6de8164))7231 min 4 s1 min 9 s2 min 12 sgsm8k exact match 0.4 on the first 5 questions

How to read it: "Steps run" counts every command, including retries after a failure. Setup time runs from the first install command to the start of the successful eval. Eval time is mostly the local model working on a CPU: Inspect's 6.6 minutes are 10 questions plus 10 grading calls, one at a time. The "first result" scores come from a 3B model on sample data. They show the tool ran, not how good the tool is.

What broke, and the fix that worked

The last column says whether the problem would also hit someone following the quickstart with a paid OpenAI key. "Setup" means it depends on the machine: system Python on recent Debian and Ubuntu refuses pip installs outside a virtual environment.

Every failure in the timed run (11 Nov 2026)
ToolWhat happenedFixHits key users too?
DeepEvalThe quickstart says to run cp .env.example .env.local. A new project has no .env.example, so the command fails.None needed for a local run; we carried on.yes
DeepEvaldeepeval set-ollama --model=qwen2.5:3b, as the linked judge docs show, reported success. The test still failed with "LocalModel is missing a required parameter".Run it again with --save=dotenv, so the setting that marks the judge as Ollama is written to .env.local.no
DeepEvalNext run: "OllamaModel requires the ollama package". The quickstart and the judge docs don't mention it.pip install ollamano
RagasThe recommended first command, uvx ragas quickstart rag_eval, crashed on import: No module named 'langchain_community.chat_models.vertexai'. Ragas 0.4.3 pulls langchain-community 0.4.2, which no longer has that module.uvx --with "langchain-community<0.4" ragas quickstart rag_evalyes
RagasSame crash inside the generated project after uv sync.uv add "langchain-community<0.4" (0.3.31 installed)yes
RagasThe Ollama tab only swaps the judge. The sample app still builds an OpenAI client and calls gpt-4o, so it stops on "Missing credentials".Point the sample app's client and model at Ollama too (two lines in evals.py and rag.py).no
Inspect AIpip install inspect-ai is refused by system Python on Debian/Ubuntu (externally managed environment) unless you are in a virtual environment. The docs don't create one.python3 -m venv .venv firstsetup
Inspect AIThe SimpleQA example stopped with "Hugging Face Datasets requires optional dependencies".pip install datasetsyes
lm-evaluation-harnessNo virtual environment step before pip install -e . (same Debian/Ubuntu issue).python3 -m venv .venv firstsetup
lm-evaluation-harnessThe README's local-completions line puts batch_size=16 inside --model_args. This version rejects it: "got multiple values for keyword argument 'batch_size'".Drop batch_size from --model_argsyes
lm-evaluation-harnessNext: No module named 'transformers'. local-completions loads a Hugging Face tokenizer named after the model, and an Ollama tag isn't one.Add tokenizer_backend=None to --model_argsno

Two smaller notes. Ragas prints a results path (experiments/) that is not where the file lands (evals/experiments/). Promptfoo exits with code 100, and DeepEval with code 1, when a test fails, so in CI the job fails even though the eval ran.

How we pointed each quickstart at a local model

Every quickstart defaults to a paid hosted model. We used each vendor's own documented local route and changed nothing else:

  • Promptfoo: Replaced the two openai: providers in promptfooconfig.yaml with ollama:chat:qwen2.5:3b (Promptfoo's own Ollama provider).
  • DeepEval: deepeval set-ollama --model=qwen2.5:3b, from DeepEval's "Configure LLM judges" docs, instead of exporting OPENAI_API_KEY.
  • Ragas: The quickstart's own "Local Models (Ollama)" snippet: an OpenAI-compatible client at localhost:11434/v1 passed to llm_factory.
  • Inspect AI: --model ollama/qwen2.5:3b instead of openai/…, plus --limit 10, so the 1,000-question SimpleQA set finishes in minutes on a CPU.
  • lm-evaluation-harness: The README's own local-server route (lm_eval[api] + local-completions) at Ollama's /v1/completions, on gsm8k with --limit 5. The README's first example needs a CUDA GPU (--device cuda:0); this machine has none.

What it means if you're picking a tool

  • Expect to debug the quickstart, not just run it. Four of five needed changes their docs don't mention. Allow about an hour per tool, not the "5 minutes" some pages promise.
  • Local-model support exists everywhere, but it's the least tested path. Half of the fixes (5 of 10) only matter once you leave the OpenAI default.
  • Pin your versions. The Ragas crash came from a new version of a dependency, not from Ragas itself. A pinned lockfile would have avoided it.
  • The quickstart is not the product. A rough first hour says nothing about metric quality, tracing or how a tool handles real datasets. For those, see Ragas vs DeepEval, DeepEval vs Promptfoo and the eval tools for product managers roundup. To start writing your own evals, see eval-driven development.

Method

Setup (recorded in logs/environment.txt)
Date and time11 Nov 2026, 07:01–07:15 IST, one tool after another
MachineDebian 13 (Linux 6.12), Intel Xeon, 8 vCPU, 15 GiB RAM, no GPU
ModelOllama 0.40.16, qwen2.5:3b (id 357c53fb659c, Q4_K_M), loaded before each tool
RuntimesPython 3.13.5 (the uv-managed Ragas project used 3.12), Node 24.22.1, npm 11.21.1, uv 0.13.7
Tool versionsWhatever each quickstart installed that morning: Promptfoo 0.125.1, DeepEval 4.3.3, Ragas 0.4.3, Inspect AI 0.3.289, lm-evaluation-harness 0.4.14.dev0 (commit d6de8164)
IsolationA fresh empty folder per tool, empty npm/pip/uv/Hugging Face caches and a new virtual environment per Python tool. API keys removed from the environment.
  1. Saved the text of each official quickstart as it read on 11 Nov 2026.
  2. Typed each command as written. Our changes are logged separately: adapt for the local model, fix for anything the docs miss.
  3. When a step failed, we kept the failure in the count, applied the smallest fix we found, and continued.
  4. Stopped the clock when the tool printed its own end-of-run summary. A run with a failing test still counts as a result.
  5. Recorded every terminal session with asciinema.

Limits

  • One timed run per tool on one machine. Seconds will differ on yours, and our box runs other jobs too. The step counts and breaks are the solid part.
  • The quickstarts don't fix temperature, so their pass/fail numbers move between runs. In earlier untimed runs, Promptfoo scored 1 of 2 and Inspect 0.0.
  • "Latest" changes. A quickstart that broke on 11 Nov 2026 may be fixed next week, or a working one may break.
  • A CPU and a 3B model make eval time long and scores low. With a hosted model, eval time would be seconds. Setup time would barely change.
  • Only the main getting-started path was used: Promptfoo's example project, DeepEval's single-turn LLM-as-a-judge tab, Ragas's uvx route, Inspect's SimpleQA example, and lm-evaluation-harness's API route rather than its GPU example.

Rerun it yourself

Everything is in one folder: a script per tool, the timing harness, raw results.csv, full logs, terminal recordings and the quickstart text we followed.

ollama pull qwen2.5:3b
tar xzf quickstart-reality-check.tar.gz && cd quickstart-reality-check
./run-all.sh        # all five, recorded; writes results.csv
./run-ragas.sh      # or one tool

FAQ

Which eval tool quickstart is easiest to run?

In our 11 Nov 2026 run, Promptfoo was the only one of five that reached a first result with no fix the docs don't mention: 3 steps and 27 seconds with a local model. That is one run on one machine, and it says nothing about which tool evaluates better.

Can you run these eval tools without an OpenAI API key?

Yes, all five reached a result with a free local model served by Ollama. Each needed its own local-model setting, and four of the five needed at least one extra fix we list on this page.

Does this test say which eval tool is best?

No. It measures only the official getting-started path: steps, breaks and time to the first result. The scores the tools printed come from a 3-billion-parameter local model on sample data, not from the tools.

How do I rerun this test?

Download the scripts, start Ollama with qwen2.5:3b, and run ./run-all.sh. It creates a fresh folder and cold caches per tool, logs every step with timings and writes results.csv.