AppWorld
by Stony Brook University NLP & collaborators
What is AppWorld? AppWorld is a benchmark for interactive coding agents: an engine of 9 everyday apps with 457 APIs, populated with the digital lives of about 100 fictional users, plus 750 tasks the agent completes by writing code against those APIs.
Editor note
A simulated world of 9 everyday apps with 457 APIs, used by about 100 fictional people, plus 750 tasks an agent solves by writing code against those APIs. State-based tests accept different valid solutions and also catch collateral damage to other data.
- What it tests
- Coding agents that operate several apps through APIs to finish daily tasks.
- Original source
- appworld.dev/
- Last updated
- Repo last commit 2026-09-04
- License
- Apache-2.0
When to use AppWorld
- Your agent works through APIs across several apps rather than one tool call at a time.
- You want tests that allow different valid solutions and also flag unwanted side effects.
Limitations
- The apps are simulated, so real-world API quirks are missing.
- Part of the code ships encrypted and is unpacked by appworld install, so follow the repo's install steps exactly.
How to run it
- pip install appworld, then appworld install and appworld download data.
- Run your agent on a split and score it with appworld evaluate.
pip install appworld
appworld install
appworld download dataNotable results
- Paper (2024): GPT-4o solved about 49% of normal tasks and about 30% of challenge tasks; other models solved at least 16% fewer. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.