MTGBench

Name: MTGBench
Creator: Yandex and Yandex School of Data Analysis (Alisa Alenicheva, Timur Zykov, Artsiom Saladukha, Vladislav Negodin)
License: https://spdx.org/licenses/MIT.html

by Yandex and Yandex School of Data Analysis (Alisa Alenicheva, Timur Zykov, Artsiom Saladukha, Vladislav Negodin)

What is MTGBench? MTGBench is a reasoning & knowledge eval made by Yandex and Yandex School of Data Analysis (Alisa Alenicheva, Timur Zykov, Artsiom Saladukha, Vladislav Negodin). Seven multi-turn text games in English and Russian: Wordle, Word Chain, Situation Puzzles, 20 Questions, Wheel of Fortune, 20 Features and Alias. Each move builds on earlier ones, so it tests whether a model uses feedback, plans ahead and stays consistent over a conversation. Four games are checked by algorithms and three by an LLM rule-enforcer that the authors validated against human annotators; episodes are written natively in each language rather than translated. The paper (Findings of EMNLP 2026) tested 16 models and a human baseline on the Russian games, and found that rankings depend heavily on the game.

Page last updated 2026-10-18 · Evals Review

Editor note

Seven multi-turn text games in English and Russian: Wordle, Word Chain, Situation Puzzles, 20 Questions, Wheel of Fortune, 20 Features and Alias. Each move builds on earlier ones, so it tests whether a model uses feedback, plans ahead and stays consistent over a conversation. Four games are checked by algorithms and three by an LLM rule-enforcer that the authors validated against human annotators; episodes are written natively in each language rather than translated. The paper (Findings of EMNLP 2026) tested 16 models and a human baseline on the Russian games, and found that rankings depend heavily on the game.

What it tests: Multi-turn reasoning, feedback use and rule consistency in word and puzzle games, in English and Russian.
Original source: github.com/skyberg11/MTGBench
Last updated: Last push 2026-10-24
License: MIT

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Related reasoning & knowledge evals

GPQA — Expert-level science reasoning.
Humanity's Last Exam — Frontier academic knowledge and reasoning.
LiveBench — Hard, contamination-limited tasks with verifiable ground-truth answers (no LLM judge) in v1; LiveBench 2.0 (14 Oct 2026) adds long-horizon real-work tasks.
MMLU — World knowledge and problem solving across many school and professional subjects.

All Reasoning & knowledge evals · Comparison table

Discussion

No one has discussed MTGBench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Be specific and kind. No promotion or affiliate links. Community rules · All discussions