# Pre-registration: re-score a small public Physics-IQ subset with the official scorer

Written 14 Nov 2026, 6:23 PM IST, before any clip download or scorer run in this test. Approved by the site owner 14 Nov 2026, 6:14 PM IST ("Run both video tests"). Cost: $0 (public data and the official Apache-2.0 scorer; CPU only; no API keys, no sign-ins, no personal accounts).

## Question
On a small, licence-cleared public subset, how closely do scores from a fresh run of the official Physics-IQ scorer match the published scores for the same clips?

## Scorer
Only the official scorer from https://github.com/google-deepmind/physics-IQ-benchmark (Apache-2.0). We will record the commit SHA. Default settings follow the repo (Physics-IQ Verified unless we pass `--original_physics_iq` because the obtainable ground truth is the original benchmark). We will not write a homemade metric and will not present any substitute as a Physics-IQ score.

## Machine and time cap
Box: 8 vCPU Intel Xeon, about 16 GB RAM, no GPU. CPU only. If the official scorer cannot finish a first clip pair in a reasonable time, or the whole planned subset would clearly exceed about 2 hours, we stop, report the time and the blocker, and do not invent a score.

## What we will score
At most 3 scenarios × 3 camera perspectives (9 clip pairs) for at most 2 published model rows (or the official two-real-takes ceiling if no licensed generated outputs are available). Generated videos must be official 5-second evaluation mp4s in the scorer's expected naming and trim. We will not run the scorer on teaser GIFs or other demo media from the project site.

## Licence and access (fixed now)
We will record the licence of every clip we use. Official real videos are CC-BY-4.0 per the repo. Generated videos need their own stated licence; a dataset card with no licence, or only "other" and no terms, is not used. We will not log into Hugging Face or any other host with a personal account. If official ground-truth videos cannot be downloaded without an account, that is a blocker: we stop and report it. A smoke test that scores two real takes against each other is allowed only when licensed generated outputs (or the published model rows they belong to) are not available, and only if the official real videos themselves can be fetched without an account. That smoke test is not a new leaderboard entry and is not a score for any named model.

## Comparison
If we obtain both official ground truth and licensed generated outputs for a published row, we compare our official-scorer output with the published score for that same setting (Verified or original, same input type, same variant). We will say how close they are, and list every clip used. If we can only run the two-real-takes smoke test, we report that number as a scorer-health check, not as a model result.

## What we will not claim
A close or distant match on a handful of clips does not confirm or refute a model's published Physics-IQ ranking. This is not a new leaderboard entry.
