Agent Initiative Benchmark
by Deepnote
What is Agent Initiative Benchmark? Agent Initiative Benchmark is an agents & tool use eval made by Deepnote. A benchmark for proactive agents: the agent reads recent Slack conversations inside PageMantle, a synthetic software company, investigates across eleven simulated services (Slack, Linear, Notion, HubSpot, Stripe, PostHog, BigQuery and others) and decides whether to step in, what to say and to whom. Some tasks are deliberately no-action, so staying silent is the correct answer. Judge models score the decision to intervene, the evidence, message quality and audience. Deepnote's v1.0 launch post (Launch YC, Nov 2026) reports 12 models on 50 tasks across eight task types; the dataset is on Hugging Face (deepnote/pagemantle-company-dataset). Results are Deepnote's own; re-read its results page before quoting ranks.
Editor note
A benchmark for proactive agents: the agent reads recent Slack conversations inside PageMantle, a synthetic software company, investigates across eleven simulated services (Slack, Linear, Notion, HubSpot, Stripe, PostHog, BigQuery and others) and decides whether to step in, what to say and to whom. Some tasks are deliberately no-action, so staying silent is the correct answer. Judge models score the decision to intervene, the evidence, message quality and audience. Deepnote's v1.0 launch post (Launch YC, Nov 2026) reports 12 models on 50 tasks across eight task types; the dataset is on Hugging Face (deepnote/pagemantle-company-dataset). Results are Deepnote's own; re-read its results page before quoting ranks.
- What it tests
- Whether an agent helps without being asked, picks the right audience, and stays silent when the problem is already handled.
- Original source
- github.com/deepnote/agent-initiative-benchmark
- Last updated
- v1.0; repo last push 2026-11-13 (GitHub API, read 16 Nov 2026)
- License
- Apache-2.0 (GitHub API, read 16 Nov 2026)
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed Agent Initiative Benchmark yet. Have you run it? Share what it caught, what it missed or how you set it up.