Agents & tool use

Taste-Bench

by City University of Hong Kong & Microsoft

What is Taste-Bench? Taste-Bench is an agents & tool use eval made by City University of Hong Kong & Microsoft. 502 two-option questions cut from real agent runs: the model sees the task and the run up to a decision point, then picks which next step the rest of the run proved better. Forks come from software-engineering runs on SWE-bench and SWE-bench Pro and from ML-research runs in METR's MALT transcripts; the paper reports the best model at 59.7%.

Page last updated 2026-10-18 · Evals Review

Editor note

502 two-option questions cut from real agent runs: the model sees the task and the run up to a decision point, then picks which next step the rest of the run proved better. Forks come from software-engineering runs on SWE-bench and SWE-bench Pro and from ML-research runs in METR's MALT transcripts; the paper reports the best model at 59.7%.

What it tests
An agent's judgement at long-horizon decision points, not just end-to-end task success.
Original source
github.com/wbopan/tastebench
Last updated
Repo last commit 2026-09-23
License
MIT

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Reviews from practitioners who sign in with GitHub open soon.

Related agents & tool use evals

  • GAIA — General assistant skills: browsing, file handling, multi-step reasoning.
  • WebArena — Autonomous web navigation and task completion.
  • OSWorld — Computer-use agents operating real operating systems and apps.
  • Terminal-Bench — Agents working in the shell end to end.

All Agents & tool use evals · Comparison table