# Our own test: does confidence routing hold up? (Evals Review, 9 Nov 2026 IST)

Draft research for the page /guide/confidence-routing-jev-vs-particle-1. Small sample. Indicative only.

## Question
If a small, fast model answers first and hands only its low-confidence cases to a bigger model,
does accuracy hold, how many cases get handed off, and is the small model's stated confidence honest
(calibrated)? This mirrors the "Jev routes unsure cases to particle-1" setup in the replication posted
by @garrytan on X on 9 Nov 2026, on public known-answer items instead of private repo decisions.

## Budget rule: free tiers only, no spend
- No card, no paid plan, no top-up. Only OpenRouter model ids ending in `:free` (enforced in run.py by an assert).
- **Jev (TypeSafe): SKIPPED.** TypeSafe's help center says it offers no trial credits, and new accounts get
  no free credit. A Vercel AI Gateway route (which lists `typesafe-ai/jev-1.13.0`) returned
  "AI Gateway requires a valid credit card on file ... to unlock your free credits" (HTTP 403), so it is not free for us.
  We could not confirm that any existing TypeSafe key we hold is on free credit, so we did not call it.
- **particle-1 (Particle): SKIPPED.** It is a paid model on OpenRouter (`particle-ai/particle-1`) and on
  Vercel AI Gateway (`particle/particle-1`), which also needed a card (same 403). Particle's own API is prepaid credit by card.
- **Stand-ins (clearly labeled, not the real models):**
  - Small "decider" stand-in for Jev: `liquid/lfm-2.5-2.6b:free` (2.6B params), asked for a yes/no answer plus a
    0-100 stated confidence (verbalized confidence; Jev instead returns token probabilities, so this is a weaker stand-in).
  - Second small stand-in (the one used on the page): Qwen2.5-1.5B-Instruct GGUF q4_k_m run locally on our CPU with
    llama-cpp-python 0.3.36 (`run_local.py`, model sha256 6a1a2eb6d15622bf3c96857206351ba97e1af16c30d7a74ee38970e434e9407e).
    Confidence = probability on the chosen yes/no first token, renormalised over yes/no tokens in the top-20 log-probs.
    This is closer to Jev's probability output than verbalized confidence. Free (local compute).
  - Big "fallback" stand-in for particle-1: `nvidia/nemotron-3-ultra-550b-a55b:free`, asked for yes/no only.
- Actual spend: $0.00 (OpenRouter reported `cost: 0` on every call; see raw/*.jsonl `cost_usd`).

## Items (frozen before any model call)
`items.jsonl`, 45 known-answer yes/no items, built by `build_items.py` with seed 20261109
(sha256 75705abff625d51154d69549f6b22cd68993ab7722c7569d2bf011dd40b55025):
- 15 x SST-2 validation sentences (stanfordnlp/sst2): "is the sentiment positive?"
- 15 x BoolQ validation passage questions (google/boolq), passage cut to 1,500 characters
- 15 x synthetic multiplication claims "a x b = c" (two-digit factors), alternating true / off by a small amount

## Procedure
1. `python3 run.py small_or` and `python3 run.py big_or`: one fresh request per item per arm, temperature 0,
   reasoning effort "low", no system prompt. Wall-clock latency measured client-side from one box in India (IST),
   so it includes network time and free-tier queueing.
2. `python3 analyze.py small_or`: accuracy per arm and task; latency p50/p95; calibration of the small model's
   stated confidence (expected calibration error, ECE, over bins <0.5, 0.5-0.6, 0.6-0.7, 0.7-0.8, 0.8-0.9,
   0.9-0.95, 0.95-1.0, the same bin edges as the chart in the replication post); and a routing sweep:
   for each threshold tau, items where the small model's confidence < tau (or it gave no parseable answer)
   are "handed off" and scored with the big model's answer for the same item.
   Routed latency = small latency + big latency on handed-off items.

## What actually ran (9 Nov 2026, ~17:40-18:00 IST)
- OpenRouter's free tier allows 50 free-model requests per day for an account with no purchased credits. On 9 Nov the cap was hit
  ("Rate limit exceeded: free-models-per-day"): big_or answered 20 of 45 items; small_or 34 of 45
  (15 SST-2, 15 BoolQ, 4 arithmetic, 1 unparseable). We did not buy credits.
- 10 Nov 2026 (~11:00 IST), after the daily reset: big_or ran the remaining 25 items (10 BoolQ, 15 arithmetic) with the same script and settings, 0 errors, cost $0. big_or now has all 45 answers. small_or was not re-run.
- small_local answered all 45.

## Headline results (python3 analyze.py small_local)
- small_local, all 45: accuracy 75.6%, mean stated confidence 96.5%, ECE 0.21, median latency 0.64 s.
  By task: SST-2 93.3% / ECE 0.065; BoolQ 80.0% / ECE 0.161; arithmetic 53.3% / ECE 0.417.
- Arithmetic: 9 answers at confidence >= 0.95, of which 3 wrong.
- big_or alone, all 45: accuracy 95.6% (SST-2 93.3%, BoolQ 93.3%, arithmetic 100%); median latency 3.29 s (p95 12.97 s).
- Routing, small_local -> big_or, all 45 (replayed answers): tau=0.95 gives 86.7% accuracy with 26.7% handed off (mean 2.43 s);
  tau=0.99 gives 91.1% with 40.0% handed off (mean 2.91 s). Big alone (95.6%) beat both.
- small_or (verbalized confidence), on the 34 items it answered: 82.4% alone, routed at tau=0.99 91.2% with 44.1% handoff; ECE 0.128.

## Limits (publish with any result)
- 45 items, one run each. One wrong answer moves a 15-item task by 6.7 points. No confidence intervals claimed.
- Stand-ins, not Jev or particle-1. Verbalized confidence from a 2.6B chat model is a different mechanism from
  Jev's calibrated probabilities, so this tests the routing *method*, not TypeSafe's or Particle's products.
- Public datasets may be in training data. Tasks are short, single-turn and text-only, unlike the agent-internal
  decisions in the replication post.
- Free-tier latency includes shared-pool queueing and is not comparable to paid-tier latency.
- Routing reuses the big arm's answer for the same item rather than a second live call (deterministic replay).

## Files
- items.jsonl: frozen inputs and gold answers
- raw/small_or.jsonl, raw/big_or.jsonl: every response (truncated raw text, parsed answer, confidence, latency, tokens, cost)
- results-small_or.json / results-small_or.csv: computed metrics
- build_items.py, run.py, analyze.py: the full method