Confidence Routing for LLMs: Jev + particle-1 Replication Checked, Plus Our Free Test (Nov 2026)
Capsule: Confidence routing means a small model answers first and hands only its low-confidence cases to a bigger model. A 9 Nov 2026 replication found that routing Jev's unsure cases to particle-1 held accuracy on 6 of 8 decisions, while particle-1 alone matched or beat that setup on 7 of 8 [@garrytan on X, 9 Nov 2026]; our free check used labeled stand-in models on all 45 public items: routing at a 0.95 threshold handed off 27% of items and scored 86.7%, while the big stand-in alone scored 95.6% (one run each, so indicative of method, not of the products). Checked 10 Nov 2026.
| Model | n | Accuracy | Handoff rate | Latency |
|---|---|---|---|---|
| Replication: Jev → particle-1 | 8 decisions | held on 6 of 8 | 9% (take grading); 64% (dream triage) | ~80 ms (Jev) |
| Replication: particle-1 alone | 8 decisions | matched or beat routing on 7 of 8 | n/a | not a single figure in the post |
| Our small stand-in (Qwen2.5-1.5B local) | 45 | 75.6% | n/a (alone) | 0.64 s median |
| Our big stand-in alone (Nemotron free) | 45 | 95.6% | n/a | 3.29 s median |
| Our routed at 0.95 | 45 | 86.7% | 27% | 2.43 s mean |
Replication rows are from the results image on the 9 Nov post [@garrytan on X, 9 Nov 2026]. Our rows use labeled stand-ins; we did not run Jev or particle-1.
How we ran this check
Free tiers only, no card. Jev and particle-1 were both skipped (no free access). Stand-ins: Qwen2.5-1.5B-Instruct local for the small model; NVIDIA Nemotron 3 Ultra on a free API for the big model (20 items on 9 Nov, the other 25 after the free daily cap reset on 10 Nov). Items: 15 SST-2, 15 BoolQ, 15 two-digit multiplication claims. Download Per-item results (CSV) and the Full method notes (METHODOLOGY).
What was claimed, and who replicated it
The original X article by @Mr_Buzzoni describes "Jev engineering": move the small yes/no and pick-one decisions inside an agent to Jev, TypeSafe's decision model, route on its calibrated confidence, and send the cases it is unsure about to a bigger model (the article uses Kimi K3) [@Mr_Buzzoni on X, 29 Sep 2026]. It quotes TypeSafe's price of $0.042 per million input tokens with output free, which matches TypeSafe's own models page [TypeSafe docs]. One line of its advice is worth keeping on its own: Set the threshold per action, not per model.
On 9 Nov 2026, @garrytan posted that his gstack-evals project tried to replicate the article and also tried particle-1, writing that it turns out this article replicates, and particle-1 is even better
[@garrytan on X, 9 Nov 2026]. particle-1 is a model from Particle, post-trained on DeepSeek V4.1 Flash to use fewer tokens [Particle].
The replication's numbers
Every number below is read from the results image attached to the replication post [@garrytan on X, 9 Nov 2026]. We did not re-run it.
| Measure | Reported result |
|---|---|
| Scope | 8 small-model decisions from gstack, gbrain and Founderstar, replayed on real inputs with known answers |
| Spend | $14 of a $60 cap; Jev itself about 10 cents |
| Jev, handing unsure cases to particle-1 | 0.2% to 7% of today's cost; accuracy held on 6 of 8; gstack's eval judges about 10 points worse; one Founderstar-style decision borderline |
| particle-1 alone | Matched or beat the routed setup on 7 of 8, at 1% to 14% of today's cost; measurably more accurate than today's model on 4 |
| Speed | Jev about 80 ms per answer, against 1 to 3 seconds |
| Where routing worked | gbrain take grading: confidence honest, only 9% handed off. Dream triage: confidence "ran backwards", 64% handed off |
| Fit of the premise | Poor for those repos: most of gstack's small decisions live in skill prompts; Founderstar makes no model calls at runtime; gbrain had already built this router and found Jev won 2 of its 9 slots |
The image's calibration chart (jev-1.13.0 on held-out splits) shows why: the take-grading panel reads ECE 0.09 on 296 held items, while the dream-triage panel reads ECE 0.49 on 109. The post puts it plainly: it only works where Jev's confidence numbers are honest.
Our own check (small, free tiers only)
We ran a small version of the same idea on public items with known answers: a small model answers yes or no with a confidence, and anything under a threshold goes to a much larger model. Full method notes and per-item results are linked in How we ran this check.
- Jev and particle-1 were not run. Our rule was free tiers only, no card. TypeSafe says it offers no trial credits [TypeSafe Help Center], and the gateways that list both models asked for a card on file. So we used stand-ins, labeled as such below. This looks at the routing method, not either product.
- Items: 45 yes/no items, frozen before any call: 15 SST-2 movie-review sentiments [SST-2 on Hugging Face], 15 BoolQ passage questions [BoolQ on Hugging Face], and 15 two-digit multiplication claims (half true, half off by a little).
- Small model (stand-in for Jev): Qwen2.5-1.5B-Instruct running locally on CPU; confidence is the probability on its first answer token. A second small stand-in (a free 2.6B model reporting a 0-100 confidence in words) is in the raw data.
- Big model (stand-in for particle-1): NVIDIA Nemotron 3 Ultra on a free API tier. It hit the free daily request cap after 20 of 45 items on 9 Nov; we ran the remaining 25 (10 BoolQ, 15 multiplication) on 10 Nov after the reset, same method, so every result below covers all 45 items.
- Cost: $0.00 actual. Latency is wall-clock from one machine in India and includes free-tier queueing.
| Our check (stand-ins) | Items | Accuracy | Mean stated confidence | ECE | Median latency |
|---|---|---|---|---|---|
| Small model, all items | 45 | 75.6% | 96.5% | 0.21 | 0.64 s |
| Small, sentiment (SST-2) | 15 | 93.3% | 98.3% | 0.07 | 0.59 s |
| Small, BoolQ | 15 | 80.0% | 96.1% | 0.16 | 1.14 s |
| Small, multiplication | 15 | 53.3% | 95.1% | 0.42 | 0.51 s |
| Big model alone | 45 | 95.6% | n/a | n/a | 3.29 s (p95 12.97 s) |
| Big model, sentiment / BoolQ / multiplication | 15 / 15 / 15 | 93.3% / 93.3% / 100% | n/a | n/a | 4.00 / 3.29 / 2.47 s |
| Routed at 0.95 threshold | 45 | 86.7% (27% handed off) | n/a | n/a | 2.43 s mean |
| Routed at 0.99 threshold | 45 | 91.1% (40% handed off) | n/a | n/a | 2.91 s mean |
What we saw, with 45 items and one run each (a small sample; one wrong answer moves a 15-item task by 6.7 points):
- On easy sentiment items the small model was right and honest (ECE 0.07), and routing changed nothing: small alone, big alone and routed all scored 93.3% on those 15 items. Routing only added latency there.
- On multiplication the small model was right about half the time while stating about 95% confidence. Of the 9 answers it gave at 95% or more, 3 were wrong, so a 0.95 threshold would have kept those errors and handed off the rest. That is our small-scale version of the replication's dream-triage result: when confidence is not honest, the threshold filters the wrong cases.
- Across all 45 items, routing at 0.95 handed off 27% and scored 86.7%; at 0.99 it handed off 40% and scored 91.1%. The big model alone scored 95.6% with no routing, so no threshold we tried beat simply sending everything to the big stand-in. The gap comes from the multiplication items, where confident wrong answers were never handed off.
- The small model was about 5 times faster at the median (0.64 s against 3.29 s) than the big one on the free tier.
When confidence routing is worth it
- Check calibration first, per decision. Hold out a few hundred known-answer cases for each decision you want to route. Plot a reliability diagram (bins of stated confidence against actual hit rate) and compute ECE. A decision with a large gap, like dream triage or our multiplication set, should not be routed on that confidence.
- Plot handoff rate against accuracy. For thresholds such as 0.5, 0.8, 0.9, 0.95 and 0.99, record how many cases go to the big model and the accuracy of the combined system. Pick the cheapest point that meets your bar.
- Compare with the big (or cheaper big) model alone. The replication's main lesson is that a cheap capable model on every case can beat a router. Include that baseline before building routing.
- Count where the decisions actually live. If they sit inside prompts or make no runtime model call, there is nothing to reroute.
- Route for speed when speed matters. Sub-100 ms answers are a real benefit in tight agent loops even when cost and accuracy are a wash.
For eval tooling that helps run these checks, see the 15-step eval checklist, our Jev listing and Jev vs Perplexity Decider.
FAQ
- Does confidence routing work?
- Only where the small model's confidence is honest. In the 9 Nov 2026 replication, routing Jev's unsure cases to particle-1 held accuracy on 6 of 8 decisions, but particle-1 alone matched or beat that setup on 7 of 8. Check calibration on your own known-answer data before you route.
- Is Jev faster than particle-1?
- In the replication, yes: about 80 milliseconds per answer for Jev against 1 to 3 seconds. The replicator calls speed Jev's real edge.
- Did Evals Review run Jev and particle-1 itself?
- No. Neither had free access for us on 9 Nov 2026, and this check was limited to free tiers. Our own check used clearly labeled open-model stand-ins to look at the routing method, not the two products.
- How do I check if a model's confidence is honest?
- Hold out a few hundred items with known answers, bin the model's stated confidence, and compare each bin's average confidence with how often it was right (a reliability diagram). Summarise the gap as expected calibration error (ECE), and plot handoff rate against accuracy for a few thresholds.
Update log
- 9 Nov 2026: first version. Replication numbers read from the 9 Nov results image; our own 45-item free-tier check with stand-in models (big-model arm stopped at 20 items by a free daily cap). Later the same day: honest title and capsule, side-by-side comparison table, and "How we ran this check" links to the CSV and METHODOLOGY (SEO Guru review).
- 10 Nov 2026: finished the big-model arm on all 45 items after the free cap reset (still $0). Routing results now cover n=45 instead of an indicative n=20: big alone 95.6%, routed at 0.95 86.7% with 27% handed off, routed at 0.99 91.1% with 40% handed off. CSV and METHODOLOGY updated.
Disclosure: no affiliate or sponsored links on this page. Evals Review has no relationship with TypeSafe, Particle or the authors of the posts cited.