WebArena vs OSWorld
WebArena or OSWorld? WebArena tests web agents on self-hosted sites (shop, forum, code hosting, content management) and checks whether the task was actually completed. OSWorld tests computer-use agents on a real Ubuntu, Windows or macOS desktop across several applications, checked by scripts.
| WebArena | OSWorld | |
|---|---|---|
| Made by | Carnegie Mellon University | XLANG Lab & collaborators |
| Environment | Self-hosted websites in a browser | Full desktop operating system in a VM |
| Checking | Functional correctness of the end state | Execution-based checking script per task |
| Headline result | Best GPT-4 agent 14.41% vs humans 78.24% | Best model 12.24% vs humans over 72.36% (369 tasks) |
| Setup | Docker or a pre-built AMI | VMware, VirtualBox, Docker with KVM, or cloud |
| Version to use | v0.2.0 or later | OSWorld-Verified (July 2025) |
| License | Apache-2.0 | — |
Verdict: when to use which
Use WebArena if your agent works only in a browser.
Use OSWorld if your agent controls a whole desktop and multiple apps.
Other comparisons: SWE-bench vs SWE-bench-Live · HumanEval vs BigCodeBench · HumanEval vs LiveCodeBench · MMLU vs GPQA · AlpacaEval vs MT-Bench (LLM-as-a-judge paper) · MT-Bench (LLM-as-a-judge paper) vs LMArena (Chatbot Arena) · BEIR vs MTEB Leaderboard · HarmBench vs StrongREJECT · GAIA vs BrowseComp