MCPHallu
by Yizhen Jiang, Haowei Guo, Tianyi Bai et al. (arXiv authors)
What is MCPHallu? MCPHallu is an agents & tool use eval made by Yizhen Jiang, Haowei Guo, Tianyi Bai et al. (arXiv authors). 358 tasks for agents working through the Model Context Protocol, run against 36 production MCP servers in isolated containers. Each task is built so that one failure type dominates: picking the wrong branch, not noticing a goal is impossible with the tools available, using a similar but wrong tool, or forgetting a value needed later. Scoring by failure type, rather than one success rate, is the point: across 14 agents the average was 0.642 out of 1, falling to 0.458 on impossible-goal tasks, where agents often keep calling tools or claim success.
Editor note
358 tasks for agents working through the Model Context Protocol, run against 36 production MCP servers in isolated containers. Each task is built so that one failure type dominates: picking the wrong branch, not noticing a goal is impossible with the tools available, using a similar but wrong tool, or forgetting a value needed later. Scoring by failure type, rather than one success rate, is the point: across 14 agents the average was 0.642 out of 1, falling to 0.458 on impossible-goal tasks, where agents often keep calling tools or claim success.
- What it tests
- Whether MCP agents hallucinate in reasoning, tool execution or memory, broken down by failure type.
- Original source
- github.com/J1zz/mcp_hallu
- Last updated
- Repo last push 2026-05-07; arXiv v1 2026-11-05
- License
- Apache-2.0
Related agents & tool use evals
- GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
- WebArena — Autonomous web navigation and task completion.
- OSWorld — Computer-use agents operating real operating systems and apps.
- Terminal-Bench — Agents working in the shell end to end.
Discussion
No one has discussed MCPHallu yet. Have you run it? Share what it caught, what it missed or how you set it up.