MMMU-Pro
by MMMU team (OSU, Waterloo & collaborators)
What is MMMU-Pro? MMMU-Pro is a harder version of the MMMU multimodal benchmark. It removes questions a text-only model can answer, adds more answer options, and introduces a vision-only setting in which the question is embedded in the image, so the model has to see and read at the same time.
Editor note
A harder rebuild of MMMU: questions a text-only model could answer were removed, more answer options were added, and a vision-only setting puts the whole question inside an image. In the paper, models scored 16.8% to 26.9%, far below their MMMU results.
- What it tests
- Multimodal reasoning when the model must read the question from the image itself.
- Original source
- arxiv.org/abs/2409.02813
- Last updated
- Repo last commit 2026-07-28
- License
- Apache-2.0
When to use MMMU-Pro
- Your model's MMMU score is high and you want a setting where text shortcuts are removed.
- You want to test reading text inside images (screenshots, photos of questions).
Limitations
- It inherits MMMU's college-exam style, so it says little about everyday visual tasks.
- Vision-only and standard settings give different scores; report which you used.
How to run it
- Load MMMU/MMMU_Pro from Hugging Face.
- Use the evaluation code in the mmmu-pro folder of the official repository.
Notable results
- Paper: models scored 16.8% to 26.9% on MMMU-Pro, substantially lower than on MMMU; OCR prompts had little effect while chain-of-thought generally helped. [Paper (arXiv)]
Sources
Reviews
No reviews yet.
Reviews from practitioners who sign in with GitHub open soon.