# Pre-registration: how much does tracing slow an LLM app down? Arize Phoenix vs MLflow, both local

Written 14 Nov 2026 IST, before any measured call. Approved by the site owner 14 Nov 2026, 3:07 PM IST ("Run the Phoenix vs MLflow tracing speed test"). Cost: $0 (pip installs, local servers, local model; no accounts, no API keys).

## Setup
- Box: 8 vCPU Intel Xeon, 16 GB RAM, shared with other jobs (recorded; noise is expected).
- Model server: Ollama 0.40.16, `qwen2.5:3b`, OpenAI-compatible endpoint http://localhost:11434/v1. The OpenAI Python client (openai 3.51.0) needs some api_key string; we pass the literal placeholder "ollama", which Ollama ignores.
- Phoenix: arize-phoenix 21.0.0 server run locally (`phoenix serve`, default SQLite storage in a fresh working dir), app instrumented with openinference-instrumentation-openai 0.1.73 via `phoenix.otel.register(...)` with its default span processor, exporting over OTLP HTTP to localhost. Separate venv.
- MLflow: mlflow 3.18.0 tracking server run locally (`mlflow server`, SQLite backend in a fresh dir), app uses `mlflow.openai.autolog()`. Separate venv.
- Baseline: the same client code, no tracing, run in each venv.

## Workload
Fixed chat call: user message "Reply with the single word OK.", temperature 0, max_tokens 5. Each condition runs in its own Python process.
Two passes, order Baseline-Phoenix-MLflow then MLflow-Phoenix-Baseline, each pass = 10 warm-up calls (discarded) + 100 measured calls per condition. So 200 measured calls per condition. Baseline runs once in each venv per pass (baseline-phx and baseline-mlf, 100 each per pass).

## Metrics
1. Primary: added latency per call = median(condition) - median(its own venv's baseline), in ms, with a bootstrap 95% CI (2,000 resamples).
2. p95 latency per condition.
3. Micro-benchmark (no network, no model): 2,000 calls of a traced no-op function per tool (Phoenix: OpenTelemetry tracer span from the registered provider; MLflow: `@mlflow.trace`), vs 2,000 untraced calls; median microseconds per call.
4. Trace visibility: after the last measured call of a condition, seconds until that trace can be read back from the local server's API.
5. Completeness: traces stored vs calls made.
6. Server resident memory (RSS) after the runs and on-disk size of each store.

## What we will not claim
Local CPU model latency (hundreds of ms or more) and a shared box make small overheads hard to separate from noise; we will say so if the CI crosses zero. Hosted Phoenix/Arize or Databricks MLflow may behave differently. This is not a test of tracing features or UI quality.

## Addendum, 14 Nov 2026 ~3:50 PM IST, written after run A and before run B
Run A (15:41-15:44 IST) completed, but during pass 2 we rendered a video on the same box for about 10 seconds (15:43:24-15:43:34), overlapping the Phoenix pass-2 condition, and graded files during pass 1. That breaks the "no other work of ours on the box" assumption. We will repeat the identical script once (run B) with nothing else of ours running, report run B as the primary result, and publish run A's numbers next to it unchanged. No other change to the method.
