WebArena-Pro
by McGill NLP (Imene Kerboua, Fatemeh Pesaran Zadeh, Xing Han Lù, Weijian Qi et al.)
What is WebArena-Pro? WebArena-Pro is an agents & tool use eval made by McGill NLP (Imene Kerboua, Fatemeh Pesaran Zadeh, Xing Han Lù, Weijian Qi et al.). A reproducible browser-agent benchmark built on 20 self-hosted web apps across six domains (productivity, communication, analytics, business operations, design and media, travel), each modelled on a well-known service. Unlike earlier WebArena-style suites, audio and video are first-class inputs, with dedicated actions for pulling information out of them, and tasks include stateful, cross-site workflows. The paper (arXiv:2611.17955) scores agents on 300 test tasks; the repo also ships 100 dev tasks and runs through BrowserGym/AgentLab, Harbor or an MCP browser service. The authors report Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tied at 62.3%, while the best generic vision-language agents reach 30.3%. These are the authors' numbers.
Editor note
A reproducible browser-agent benchmark built on 20 self-hosted web apps across six domains (productivity, communication, analytics, business operations, design and media, travel), each modelled on a well-known service. Unlike earlier WebArena-style suites, audio and video are first-class inputs, with dedicated actions for pulling information out of them, and tasks include stateful, cross-site workflows. The paper (arXiv:2611.17955) scores agents on 300 test tasks; the repo also ships 100 dev tasks and runs through BrowserGym/AgentLab, Harbor or an MCP browser service. The authors report Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 tied at 62.3%, while the best generic vision-language agents reach 30.3%. These are the authors' numbers.
- What it tests
- Whether web agents can finish multi-step, multimodal (text, image, audio, video) tasks across realistic self-hosted apps, reproducibly.
- Original source
- webarena-pro.github.io
- Last updated
- arXiv:2611.17955 (13 Nov 2026); repo last push 2026-11-16 (GitHub API, read 17 Nov 2026)
- License
- MIT (GitHub API, read 17 Nov 2026)
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed WebArena-Pro yet. Have you run it? Share what it caught, what it missed or how you set it up.