Physics-IQ Verified
by Anates Labs with University of Technology Nuremberg, Tübingen AI Center, TU Munich / Helmholtz AI, and Google DeepMind (Tim Rädsch and collaborators, including Physics-IQ authors Priyank Jaini and Robert Geirhos)
What is Physics-IQ Verified? Physics-IQ Verified is an image & vision eval made by Anates Labs with University of Technology Nuremberg, Tübingen AI Center, TU Munich / Helmholtz AI, and Google DeepMind (Tim Rädsch and collaborators, including Physics-IQ authors Priyank Jaini and Robert Geirhos). A June 2026 audit of Physics-IQ that refined 57.6% of samples and more than 34.8% of prompts, and introduced sample-level scoring that weights each sample and metric equally. The authors recommend 4-run averages for SOTA claims. The official table lives in the physics-IQ-benchmark README and at physics-iq-verified.anates.ai. The Hugging Face dataset (Anates-Labs-Research/Physics-IQ-Verified) is CC-BY-4.0 and gated (access requests auto-approved, but an unauthenticated download returned HTTP 401 on 14 Nov 2026).
Editor note
A June 2026 audit of Physics-IQ that refined 57.6% of samples and more than 34.8% of prompts, and introduced sample-level scoring that weights each sample and metric equally. The authors recommend 4-run averages for SOTA claims. The official table lives in the physics-IQ-benchmark README and at physics-iq-verified.anates.ai. The Hugging Face dataset (Anates-Labs-Research/Physics-IQ-Verified) is CC-BY-4.0 and gated (access requests auto-approved, but an unauthenticated download returned HTTP 401 on 14 Nov 2026).
- What it tests
- A tightened continuation-vs-reality physics score, with audited prompts and labels.
- Original source
- arxiv.org/abs/2606.18943
- Last updated
- arXiv:2606.18943 (June 2026); official repo last push 2026-10-20; HF card licence read 14 Nov 2026
- License
- CC-BY-4.0 (Hugging Face dataset card, read 14 Nov 2026); scorer Apache-2.0
Related image & vision evals
Discussion
No one has discussed Physics-IQ Verified yet. Have you run it? Share what it caught, what it missed or how you set it up.