Image & vision

Code2Video-Compass

by Fudan University and Sun Yat-sen University (Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, Siming Chen)

What is Code2Video-Compass? Code2Video-Compass is an image & vision eval made by Fudan University and Sun Yat-sen University (Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, Siming Chen). A general-purpose benchmark for code-based video generation: a model must write executable programs that produce a complete video, across text-to-video, image-to-video and video-to-video settings. It has 2,104 tasks in 10 categories and 24 subtypes, scored with 12 objective metrics and five subjective metrics (including an MLLM judge). Paper: arXiv:2610.28416 (20 Oct 2026). Code is at github.com/code2video-compass/Code2Video-Compass.

Page last updated 2026-10-18 · Evals Review

Editor note

A general-purpose benchmark for code-based video generation: a model must write executable programs that produce a complete video, across text-to-video, image-to-video and video-to-video settings. It has 2,104 tasks in 10 categories and 24 subtypes, scored with 12 objective metrics and five subjective metrics (including an MLLM judge). Paper: arXiv:2610.28416 (20 Oct 2026). Code is at github.com/code2video-compass/Code2Video-Compass.

What it tests
Whether a model can write code that builds a complete, aligned T2V, I2V or V2V video.
Original source
code2video-compass.github.io/
Last updated
arXiv:2610.28416 (2026-10-20); GitHub repo last push 2026-11-11; LICENSE read 14 Nov 2026
License
Apache-2.0 (GitHub LICENSE, read 14 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related image & vision evals

  • MMMU — Expert-level multimodal understanding and reasoning.
  • MathVista — Mathematical reasoning over figures, plots and images.
  • MMBench — Broad perception and reasoning skills of vision-language models.
  • ChartQA — Question answering over charts.

All Image & vision evals · Comparison table

Discussion

No one has discussed Code2Video-Compass yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions