Odysseys2
by Carnegie Mellon University and Browser Use (Jang, Yue, Müller, Zunic et al.)
What is Odysseys2? Odysseys2 is an agents & tool use eval made by Carnegie Mellon University and Browser Use (Jang, Yue, Müller, Zunic et al.). A web-agent benchmark of 100 long tasks built from the real distribution of requests sent to Browser Use, run on the live web and usually asking for a deliverable such as a spreadsheet or written comparison. Each task has 8 to 12 rubric items (982 in all). Its LLM judge reads the delivered files alongside the action log and screenshots, and scores an attempt zero if it flags suspected fabrication or reward hacking. The authors compare 15 agents and 3 harnesses and release all tasks, rubrics and scores for download.
Editor note
A web-agent benchmark of 100 long tasks built from the real distribution of requests sent to Browser Use, run on the live web and usually asking for a deliverable such as a spreadsheet or written comparison. Each task has 8 to 12 rubric items (982 in all). Its LLM judge reads the delivered files alongside the action log and screenshots, and scores an attempt zero if it flags suspected fabrication or reward hacking. The authors compare 15 agents and 3 harnesses and release all tasks, rubrics and scores for download.
- What it tests
- Long, real-world browsing tasks with verifiable file deliverables.
- Original source
- odysseysbench.com/v2/
- Last updated
- Released October 2026
- License
- Not checked yet — see source
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed Odysseys2 yet. Have you run it? Share what it caught, what it missed or how you set it up.