Agents & tool use

TuringSpace

by Xuwei Ding, Skylar Zhai, Linxin Song et al.

What is TuringSpace? TuringSpace is an agents & tool use eval made by Xuwei Ding, Skylar Zhai, Linxin Song et al.. A browser environment of 125 interactive CAPTCHA-style puzzles that test spatial reasoning through action: the model sees a full-screen screenshot, clicks and drags, looks at the result and presses Verify when it believes the puzzle is solved. Tasks are split evenly across five abilities (spatial relations, closed-loop manipulation, multi-step planning, mental rotation or folding, and tracking objects over frames). Every task is a program that generates fresh instances from a seed together with a checker, so the same tasks double as a training gym; the benchmark runs each on three fixed seeds and reports pass@1 and pass^3. The paper (arXiv:2611.25387) reports people solving about 94% of tasks while the best model tested still fails about a quarter, with most errors traced to misreading the screenshot.

Page last updated 2026-10-18 · Evals Review

Editor note

A browser environment of 125 interactive CAPTCHA-style puzzles that test spatial reasoning through action: the model sees a full-screen screenshot, clicks and drags, looks at the result and presses Verify when it believes the puzzle is solved. Tasks are split evenly across five abilities (spatial relations, closed-loop manipulation, multi-step planning, mental rotation or folding, and tracking objects over frames). Every task is a program that generates fresh instances from a seed together with a checker, so the same tasks double as a training gym; the benchmark runs each on three fixed seeds and reports pass@1 and pass^3. The paper (arXiv:2611.25387) reports people solving about 94% of tasks while the best model tested still fails about a quarter, with most errors traced to misreading the screenshot.

What it tests
Spatial reasoning by computer-use agents acting on interactive screen puzzles.
Original source
github.com/os-cua-framework/TuringSpace
Last updated
arXiv:2611.25387; repo last pushed 2026-11-20 (GitHub API, read 21 Nov 2026)
License
Apache-2.0 (code, GitHub API and README, read 21 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related agents & tool use evals

  • GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
  • WebArena — Autonomous web navigation and task completion.
  • OSWorld — Computer-use agents operating real operating systems and apps.
  • Terminal-Bench — Agents working in the shell end to end.

All Agents & tool use evals · Comparison table

Discussion

No one has discussed TuringSpace yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions