Image and vision benchmarks, compared
Vision benchmarks test whether a model can read and reason over images: expert diagrams, math figures, charts and general perception.
| Eval | Made by | What it tests | License | Last updated | Source |
|---|---|---|---|---|---|
| MMMU | MMMU team (OSU, Waterloo & collaborators) | Expert-level multimodal understanding and reasoning. | Apache-2.0 | Repo last commit 2026-07-28 | mmmu-benchmark.github.io |
| MathVista | UCLA, University of Washington & Microsoft Research | Mathematical reasoning over figures, plots and images. | CC-BY-SA-4.0 | Repo last commit 2026-07-27 | mathvista.github.io |
| MMBench | OpenCompass (Shanghai AI Lab & collaborators) | Broad perception and reasoning skills of vision-language models. | Apache-2.0 | Repo last commit 2025-05-22 | github.com |
| ChartQA | Masry et al. (academic) | Question answering over charts. | GPL-3.0 | Repo last commit 2025-04-18 | github.com |
In short
MMMU — About 11.5K college-level questions across six disciplines and 30 subjects, each pairing text with images such as charts, diagrams, maps and chemical structures.
MathVista — 6,141 math problems set in visual contexts, drawn from 28 existing multimodal datasets plus three new ones (IQTest, FunctionQA, PaperQA).
MMBench — A bilingual (English and Chinese) multiple-choice benchmark that breaks vision-language ability into fine-grained skills, with a scoring method meant to be more robust than single-pass answers.
ChartQA — 9.6K human-written and 23.1K generated questions about real charts that need visual reading plus logical and arithmetic reasoning.
Browse the full Image & vision evals category, or see other comparisons.