# Pre-registration: human-preference Elo vs physics scores on published video-generation boards

Written 14 Nov 2026, 6:23 PM IST, before any leaderboard snapshot in this test. Approved by the site owner 14 Nov 2026, 6:14 PM IST ("Run both video tests"). Cost: $0 (public pages only; no API keys, no sign-ins, no personal accounts).

## Question
On the public tables we can read on the snapshot date, how well do human-preference ranks (Elo or equivalent) agree with physics or physical-commonsense scores for the same video-generation models? We will report rank correlation and the biggest rank movers among unambiguously matched models.

## Honest claim
We compared published leaderboards on the snapshot date. We did not run any video model and did not re-score any video in this test.

## Sources we will try to read (public HTTP only)
1. Artificial Analysis Video Arena, text-to-video board: https://artificialanalysis.ai/video/arena and the T2V leaderboard page linked from it (AA-Video-T2V). Use the official on-page table or the structured payload that renders that table. Record the board version name if shown.
2. LMArena Video: https://arena.ai/leaderboard/video and the text-to-video board linked from it, if readable.
3. Physics-IQ and Physics-IQ Verified official tables: https://github.com/google-deepmind/physics-IQ-benchmark (README), https://physics-iq.github.io/, and the official Verified page / README linked from that repo. The official score column is the headline Physics-IQ (or Verified) score. A "net improvement vs track mean" column is not the headline score and will not be used as one.
4. VBench-2.0 official Total Score: https://vchitect.github.io/VBench-2.0-project/ and the official Hugging Face Space or dataset named by the Vchitect project. We will not invent a total from unofficial copies.
5. VideoPhy-2 official hard-subset joint score: https://videophy2.github.io/ (and the official paper table if the page does not list numbers).

Each response is saved under `public/data/tests/2026-11-14-human-votes-vs-physics-scores/raw/` with the requested URL, UTC and IST retrieval timestamps, HTTP status and response body.

## Dropped boards
If a board is unreadable (Cloudflare challenge, HTTP 401/403, login wall, or no official numeric table), we record the failure and drop that board from the correlation. We do not paste numbers from memory, screenshots we cannot store, or third-party mirrors.

## Model matching (fixed now)
A human-board row and a physics-board row are a match only when all of the following hold:

1. Same model family and the same version token (for example Veo 3.1 Lite matches Veo 3.1 Lite; it does not match Veo 3.1, Veo 3.1 Fast, or an unversioned "Veo").
2. Same input type. The text-to-video human board is compared only with T2V or similarly prompted rows. Image-to-video and video-to-video / multiframe rows stay unmatched. We will not collapse i2v, t2v and v2v into one row.
3. Best-of-N, reward-model, prompt-rewriter or other add-on runs are different rows from the plain model and stay unmatched to the plain model.
4. If a name is ambiguous after that (for example "Grok Imagine Video" vs "Grok Imagine Video 1.5"), it stays unmatched.
5. If one human row could match more than one physics row, or the reverse, both stay unmatched and we note the collision.

We will publish the mapping we used, including unmatched rows and the reason.

## Metrics (fixed now)
For each readable human board × readable physics board:

- If there are 4 or more matched pairs: Spearman rank correlation of the published scores, Kendall's tau-b, and the five largest absolute rank gaps (human rank minus physics rank on the matched subset).
- If there are 6 or more pairs: a 2,000-resample bootstrap 95% interval on Spearman rho.
- If there are fewer than 4 matches: we publish the tables and the unmatched list and do not compute a correlation.

We will not invent a VBench-2.0 total or any other missing official number.

## What we will not claim
This is not a new leaderboard and not a test of any one model. Board composition, prompt style, input type and date all differ. A low (or undefined) correlation can mean the boards are not measuring the same thing, not that either board is "wrong".
