Official Physics-IQ scorer on a small public subset
Short answer: On 14 Nov 2026 we ran the official Physics-IQ scorer (Apache-2.0, commit d3f3de1) on a CPU. We could not re-score any published model row, because the official Physics-IQ Verified ground-truth videos are on a gated Hugging Face dataset and our pre-registration forbids logging in with a personal account. What we could do is a scorer-health check on the original benchmark's public videos: for 3 scenarios and 3 camera views (9 pairs), we treated the second real 5-second take as the generated video. The official original formula scored 100.0 on that subset, and the official view formula scored 100.0. Mask generation took 68 seconds and scoring took 142.9 seconds. This is not a new leaderboard entry and is not comparable to a published 66-scenario score.
Limits: read these before quoting a number
- Not a published model score. We did not score a named generator. The 100.0 figure is the official formula on two real takes, over 3 scenarios only.
- Not the full benchmark. The official command-line entry point requires 198 generated take-1 videos. We did not edit that program. We called the official metric functions on the pre-registered 9-pair subset.
- We could not reach Verified ground truth without an account, so we could not compare our run with a published Verified model row (for example Kandinsky-WM 1.0, whose generated videos are Apache-2.0).
- A 100 on two real takes is the expected ceiling of a formula that is normalised against the second real take. It shows the official code ran, not that a generator is strong.
Method (pre-registered)
The method was committed before any clip was downloaded: PREREGISTRATION.md (git commit eb28592, 2026-11-14 18:23:11 IST).
| Scorer | Official package from google-deepmind/physics-IQ-benchmark at commit d3f3de1. Functions used: generate_binary_masks (for the stand-in generated clips only), process_videos with n_processes=0 and video_time_selection="first", calculate_iq_score and IQTable.from_csv. We did not write a substitute metric. |
|---|---|
| Clips | Official original 5-second testing videos and official real masks from the public GCS bucket physics-iq-benchmark, licence CC-BY-4.0. Scenarios: ball-and-block-fall, ball-hits-duck, ball-hits-nothing. Each file is 30 FPS, 150 frames, 5.000 seconds. SHA-256 and retrieval times are in the source log. |
| What was "generated" | The official take-2 testing video for the same scenario and camera, renamed to the take-1 generated filename the scorer expects. That is the official two-real-takes ceiling check, not a model. |
| Machine | 8 vCPU Intel Xeon, about 16 GB RAM, no GPU, n_processes=0. |
Results
| Quantity | Value |
|---|---|
Official original-formula score (calculate_iq_score and IQTable.final_score_orig) | 100.0 |
Official view formula (IQTable.final_score_view) | 100.0 |
Physical variance returned by calculate_iq_score | 1.6117 |
| Official mask generation for the 9 stand-in clips | 68 seconds |
Official process_videos | 142.9 seconds |
| Total of those two steps | 210.9 seconds |
We started with a different third scenario, ball-behind-rotating-paper, and then set it aside: in the official public bucket, the take-1 and take-2 testing videos are byte-identical on all three views. Scoring that scenario would have been an identity check, not a two-takes check. We replaced it with ball-hits-nothing, whose take-1 and take-2 files differ, before running the official metric functions.
Why we did not re-score a published model
- The official Physics-IQ Verified dataset on Hugging Face (Anates-Labs-Research/Physics-IQ-Verified) is CC-BY-4.0 but gated. An unauthenticated README fetch returned HTTP 401.
- Kandinsky-WM 1.0 generated videos are Apache-2.0 and public, but they are Verified submissions and need that gated ground truth.
- Two other public generated-video datasets linked from the official README were not used: the deepkyu Cosmos3 cards state no licence, and GrainSack WPM-PhysicsIQ is licence "other" with no terms on the card.
- The official original benchmark's videos are public on GCS, which is why the two-real-takes check could run at all.
Credit
Physics-IQ is by INSAIT, Sofia University and Google DeepMind. The official scorer and the original filmed videos are published by Google DeepMind in physics-IQ-benchmark. Physics-IQ Verified is by Anates Labs with University of Technology Nuremberg, Tübingen AI Center, TU Munich / Helmholtz AI, and Google DeepMind. Software is Apache-2.0; the official videos we used are CC-BY-4.0.
Data, scripts and licence
Download the pre-registration, source and licence log, GCS download manifest with SHA-256, the scorer log, metrics and per-scenario CSV. The official licence text is in official-LICENSE.txt. Video binaries were not committed (size); they can be re-fetched from the hashed URLs.
Licence: our method notes, logs, hashes and this page text are CC BY 4.0. Attribution: Evals Review editorial team, Official Physics-IQ scorer on a small public subset, 14 Nov 2026, https://evalsreview.com/tests/physics-iq-public-sample-rescore. The scorer is Apache-2.0; the official original videos are CC-BY-4.0 (Google DeepMind / Physics-IQ authors).
Related
- Human video Elo vs physics scores (published tables, same day).
- Physics-IQ and Physics-IQ Verified in our catalog.
- All hands-on tests.