Image & vision

EvalCrafter

by EvalCrafter authors (CVPR 2024)

What is EvalCrafter? EvalCrafter is an image & vision eval made by EvalCrafter authors (CVPR 2024). An early large-model video-generation evaluation suite (CVPR 2024) covering visual quality and basic temporal behaviour over a prompt set of about 700 examples (as later compared on the WorldScore project page). Official code: github.com/evalcrafter/EvalCrafter. The repo had no LICENSE file and last push 3 Oct 2024 when checked on 14 Nov 2026; later suites such as VBench-2.0 and StreamEval now cover more of this ground.

Page last updated 2026-10-18 · Evals Review

Editor note

An early large-model video-generation evaluation suite (CVPR 2024) covering visual quality and basic temporal behaviour over a prompt set of about 700 examples (as later compared on the WorldScore project page). Official code: github.com/evalcrafter/EvalCrafter. The repo had no LICENSE file and last push 3 Oct 2024 when checked on 14 Nov 2026; later suites such as VBench-2.0 and StreamEval now cover more of this ground.

What it tests
Quality and temporal behaviour of large video generation models.
Original source
evalcrafter.github.io/
Last updated
Repo last push 2024-10-03 (GitHub API, earlier read 14 Nov 2026); LICENSE still absent 14 Nov 2026
License
Not checked yet — see source

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related image & vision evals

  • MMMU — Expert-level multimodal understanding and reasoning.
  • MathVista — Mathematical reasoning over figures, plots and images.
  • MMBench — Broad perception and reasoning skills of vision-language models.
  • ChartQA — Question answering over charts.

All Image & vision evals · Comparison table

Discussion

No one has discussed EvalCrafter yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions