AutomationBench
by Zapier (Daniel Shepard, Robin Salimans)
What is AutomationBench? AutomationBench is an agents & tool use eval made by Zapier (Daniel Shepard, Robin Salimans). A business-workflow benchmark from Zapier. Each task starts a simulated company environment (CRM, calendar, inbox and more) across 47 simulated SaaS tools; the agent must find the right REST endpoints itself, follow layered business policies, ignore irrelevant or misleading records and leave every system in the correct end state. The public set has 600 tasks, 100 each in sales, marketing, operations, support, finance and HR, plus 200 simple warm-up tasks that do not count. Grading is programmatic and strict: a task passes only if every end-state assertion passes, and the best public-set pass rate in the repo is 50.3% (Claude Opus 5). The official leaderboard on zapier.com/benchmarks uses a harder held-out private set, and Artificial Analysis runs its own version, AutomationBench-AA.
Editor note
A business-workflow benchmark from Zapier. Each task starts a simulated company environment (CRM, calendar, inbox and more) across 47 simulated SaaS tools; the agent must find the right REST endpoints itself, follow layered business policies, ignore irrelevant or misleading records and leave every system in the correct end state. The public set has 600 tasks, 100 each in sales, marketing, operations, support, finance and HR, plus 200 simple warm-up tasks that do not count. Grading is programmatic and strict: a task passes only if every end-state assertion passes, and the best public-set pass rate in the repo is 50.3% (Claude Opus 5). The official leaderboard on zapier.com/benchmarks uses a harder held-out private set, and Artificial Analysis runs its own version, AutomationBench-AA.
- What it tests
- Whether AI agents can complete real cross-app business workflows through APIs while following company policy.
- Original source
- github.com/zapier/AutomationBench
- Last updated
- Paper arXiv 2604.18934 (21 Apr 2026); repo last push 2026-08-04
- License
- MIT (LICENSE file in zapier/AutomationBench)
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed AutomationBench yet. Have you run it? Share what it caught, what it missed or how you set it up.