Image & vision

Physics-IQ Verified

by Anates Labs with University of Technology Nuremberg, Tübingen AI Center, TU Munich / Helmholtz AI, and Google DeepMind (Tim Rädsch and collaborators, including Physics-IQ authors Priyank Jaini and Robert Geirhos)

What is Physics-IQ Verified? Physics-IQ Verified is an image & vision eval made by Anates Labs with University of Technology Nuremberg, Tübingen AI Center, TU Munich / Helmholtz AI, and Google DeepMind (Tim Rädsch and collaborators, including Physics-IQ authors Priyank Jaini and Robert Geirhos). A June 2026 audit of Physics-IQ that refined 57.6% of samples and more than 34.8% of prompts, and introduced sample-level scoring that weights each sample and metric equally. The authors recommend 4-run averages for SOTA claims. The official table lives in the physics-IQ-benchmark README and at physics-iq-verified.anates.ai. The Hugging Face dataset (Anates-Labs-Research/Physics-IQ-Verified) is CC-BY-4.0 and gated (access requests auto-approved, but an unauthenticated download returned HTTP 401 on 14 Nov 2026).

Page last updated 2026-10-18 · Evals Review

Editor note

A June 2026 audit of Physics-IQ that refined 57.6% of samples and more than 34.8% of prompts, and introduced sample-level scoring that weights each sample and metric equally. The authors recommend 4-run averages for SOTA claims. The official table lives in the physics-IQ-benchmark README and at physics-iq-verified.anates.ai. The Hugging Face dataset (Anates-Labs-Research/Physics-IQ-Verified) is CC-BY-4.0 and gated (access requests auto-approved, but an unauthenticated download returned HTTP 401 on 14 Nov 2026).

What it tests
A tightened continuation-vs-reality physics score, with audited prompts and labels.
Original source
arxiv.org/abs/2606.18943
Last updated
arXiv:2606.18943 (June 2026); official repo last push 2026-10-20; HF card licence read 14 Nov 2026
License
CC-BY-4.0 (Hugging Face dataset card, read 14 Nov 2026); scorer Apache-2.0

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related image & vision evals

  • MMMU — Expert-level multimodal understanding and reasoning.
  • MathVista — Mathematical reasoning over figures, plots and images.
  • MMBench — Broad perception and reasoning skills of vision-language models.
  • ChartQA — Question answering over charts.

All Image & vision evals · Comparison table

Discussion

No one has discussed Physics-IQ Verified yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions