Image & vision

RoughCut-Bench

by Ruisi Zhao, You Li, Sihan Chen et al. (Zhejiang University ReLER and ByteDance)

What is RoughCut-Bench? RoughCut-Bench is an image & vision eval made by Ruisi Zhao, You Li, Sihan Chen et al. (Zhejiang University ReLER and ByteDance). Tests whether vision-language models can make the first rough cut of a video from raw footage. In the visual setting (101 cases built from 831 source videos and images, with 1,476 professional highlight annotations) the model must pick highlights, order the source clips and, in some cases, follow a user's brief. In the speech setting (84 ASR transcripts) it must cut filler words, repeated expressions and low-value passages. Cut plans are scored against the annotations, and rendered visual cuts are also rated by an agentic evaluator the authors aligned with human judgement. The repository includes the agentic workflows used to turn models into cut planners; the paper (arXiv:2611.24098, KDD 2027) reports that current models still fall short of expert-aligned cut plans.

Page last updated 2026-10-18 · Evals Review

Editor note

Tests whether vision-language models can make the first rough cut of a video from raw footage. In the visual setting (101 cases built from 831 source videos and images, with 1,476 professional highlight annotations) the model must pick highlights, order the source clips and, in some cases, follow a user's brief. In the speech setting (84 ASR transcripts) it must cut filler words, repeated expressions and low-value passages. Cut plans are scored against the annotations, and rendered visual cuts are also rated by an agentic evaluator the authors aligned with human judgement. The repository includes the agentic workflows used to turn models into cut planners; the paper (arXiv:2611.24098, KDD 2027) reports that current models still fall short of expert-aligned cut plans.

What it tests
Video rough-cut planning: highlight selection, clip ordering, brief adherence and transcript cleanup.
Original source
github.com/Whalesong-zrs/RoughCut-Bench
Last updated
arXiv:2611.24098; repo and HF dataset zhaors00/RoughCut-Bench last updated 2026-11-19 (GitHub and HF APIs, read 20 Nov 2026)
License
Apache-2.0 (code, GitHub API; dataset, HF tag; read 20 Nov 2026)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related image & vision evals

  • MMMU — Expert-level multimodal understanding and reasoning.
  • MathVista — Mathematical reasoning over figures, plots and images.
  • MMBench — Broad perception and reasoning skills of vision-language models.
  • ChartQA — Question answering over charts.

All Image & vision evals · Comparison table

Discussion

No one has discussed RoughCut-Bench yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions