# Quickstart reality check (evalsreview.com test #1)

**Question:** if you follow each eval tool's official quickstart exactly as written today, how many steps does it take to get a first eval result, what breaks, and how long does it take on a laptop-class machine with no paid API key?

Tools: Promptfoo, DeepEval, Ragas, Inspect AI, EleutherAI lm-evaluation-harness.
Run on 11 Nov 2026 (IST). Results page: https://evalsreview.com/tests/quickstart-reality-check (draft until published).

## What this measures, and what it doesn't

- It measures the official getting-started path: the number of steps, the steps that failed or were missing, the changes needed to use a local model, and wall-clock time from the first install command to the first eval result.
- It does **not** measure tool quality, metric accuracy, scale or production fitness. The scores the tools print come from a 3B local model on each quickstart's sample data. They are not scores of the tools.

## Method

1. **Same machine, same model, back to back.** One Linux box: Debian 13, 8 vCPU, 15 GiB RAM, no GPU. One local model, Ollama `qwen2.5:3b` (exact id and digests in `logs/environment.txt`). The model is loaded before each tool, so none of them pays the load time.
2. **Fresh start per tool.** Each tool gets an empty folder (`$WORK_ROOT/<tool>`, default `/tmp/qrc-work`) and empty npm/pip/uv/Hugging Face caches, so every package is downloaded as a first-time user would download it. Every `*API_KEY*`, `*TOKEN*` and `*SECRET*` variable is removed from the environment first, so nothing can fall back to a paid model.
3. **Follow the docs as written.** The text of each quickstart, as fetched on 11 Nov 2026, is in `sources/`. Each command the script types is logged as one of three kinds of step:
   - `official`: copied from the quickstart.
   - `adapt`: our change to use the local Ollama model instead of the paid default, using the vendor's own documented local route.
   - `fix`: a step the quickstart didn't mention but that was needed to continue.
   When an official step fails, the script records the failure, applies the smallest fix we found, and carries on. Failures stay in the count.
4. **Timing.** `total_seconds_to_first_result` runs from the start of the first command to the end of the command that printed the first eval result. `final_eval_seconds` is the eval itself, which on a CPU depends mostly on the model. `setup_seconds` is everything before it. One timed run per tool.
5. **What counts as a first result.** The tool printed its own "finished" summary (pass/fail table, accuracy or score file). This counts even if a test inside the run failed: Promptfoo exits 100 and pytest-based DeepEval exits 1 when a metric fails, and both still produced a result.
6. **Caps on long evals.** Inspect's SimpleQA example is capped at the first 10 questions (`--limit 10`) and lm-eval-harness's gsm8k at 5 (`--limit 5`), so a CPU-only box finishes in minutes. The full sets are about 1,000 and 1,319 questions.

## Rerun it

You need Ollama serving the model on `localhost:11434` (`ollama pull qwen2.5:3b`), plus Python 3.10+, Node 18+/npm, `uv`, `git` and `asciinema`.

```bash
git clone https://github.com/<repo>/evalsreview  # or download this folder
cd tests/quickstart-reality-check
./run-all.sh                  # all five, recorded; writes results.csv
./run-ragas.sh                # or any single tool (logs only, no recording)
```

`MODEL_TAG=llama3.2:3b ./run-all.sh` swaps the model. Expect different seconds on different hardware. The step counts and failures should match as long as the tools' latest versions haven't changed.

## Files

| Path | What |
|---|---|
| `run-<tool>.sh` | The exact steps per tool, labelled official / adapt / fix |
| `lib.sh` | Fresh folder, cold caches, key scrubbing, step timer |
| `run-all.sh` | Runs all five under asciinema, then `build-results.py` |
| `results.csv` | One row per tool, raw numbers |
| `logs/<tool>.steps.tsv` | Per-step start/end epoch, seconds and exit code |
| `logs/<tool>.log` | Full terminal output of the timed run |
| `logs/<tool>.version` | Versions that actually ran |
| `logs/environment.txt` | OS, CPU, RAM, Python/Node/uv/Ollama versions, model id and digests |
| `logs/exploration/` | Our first attempts, kept so the failures can be checked |
| `casts/<tool>.cast` | asciinema recordings of the timed runs (play with `asciinema play`) |
| `sources/` | Text of each official quickstart as fetched on 11 Nov 2026 |

## How the model was installed

On this network `ollama pull qwen2.5:3b` failed: Ollama refused a download redirect that resolved to a private address. We fetched the same manifest and blobs from `registry.ollama.ai` with curl, checked each blob's sha256 against the manifest, and placed them in `~/.ollama/models`. `ollama list` shows id `357c53fb659c`, the official `qwen2.5:3b` tag.

## Notes on the 11 Nov 2026 run

- lm-eval-harness: the retry without `batch_size` in `--model_args` was logged as an `adapt` step in that run's `steps.tsv`. It is really a fix, because the README line itself fails, and the script now labels it `fix`. The results page counts it as a fix. The raw logs are left as recorded.
- Ragas prints `Experiment results saved to: …/rag_eval/experiments/<name>.csv`, but the file is written to `rag_eval/evals/experiments/`. Our copy is `logs/ragas.experiment.csv`.
- The pass/fail numbers move between runs, because the quickstarts don't fix temperature. In our exploration runs (`logs/exploration/`), Promptfoo scored 1 of 2, DeepEval passed and Inspect scored 0.0 on 10 questions. The timed run is what we report.
