SAB (Startup Agent Benchmark)
by Mento.vc (Kalpin, Zhuravlev, Golomidov, Bilich, Li)
What is SAB (Startup Agent Benchmark)? SAB (Startup Agent Benchmark) is an agents & tool use eval made by Mento.vc (Kalpin, Zhuravlev, Golomidov, Bilich, Li). A venture-capital research benchmark from the fund Mento. Agents answer 960 analyst tasks in six workflows (including sourcing screens, product, market and why-now, competition and traction) using a task-scoped search and fetch tool over a corpus with stale and irrelevant documents, half built on synthetic companies and half on a frozen snapshot of Mento's data. Each task is scored out of 100 for answer content and cited evidence, with a strict pass rule that fails any made-up source, and a utility score that rewards answering 'unknown' when the needed fact was deliberately removed. A second part asks models to rank cohorts of 100 companies, scored against their later web-traffic and headcount growth. In the October 2026 results for 14 models, the best research score was 97.2 of 100, every model still gave values for 9% to 33% of withheld facts, and all 13 models that finished the ranking cohorts scored below a simple past-headcount-growth rule.
Editor note
A venture-capital research benchmark from the fund Mento. Agents answer 960 analyst tasks in six workflows (including sourcing screens, product, market and why-now, competition and traction) using a task-scoped search and fetch tool over a corpus with stale and irrelevant documents, half built on synthetic companies and half on a frozen snapshot of Mento's data. Each task is scored out of 100 for answer content and cited evidence, with a strict pass rule that fails any made-up source, and a utility score that rewards answering 'unknown' when the needed fact was deliberately removed. A second part asks models to rank cohorts of 100 companies, scored against their later web-traffic and headcount growth. In the October 2026 results for 14 models, the best research score was 97.2 of 100, every model still gave values for 9% to 33% of withheld facts, and all 13 models that finished the ranking cohorts scored below a simple past-headcount-growth rule.
- What it tests
- Whether research agents find and cite the right evidence, say 'unknown' instead of inventing facts, and rank companies better than a simple heuristic.
- Original source
- mento.vc/sab-bench
- Last updated
- Results October 2026 (answer keys v0.7.1)
- License
- Not checked yet — see source
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed SAB (Startup Agent Benchmark) yet. Have you run it? Share what it caught, what it missed or how you set it up.