Image & vision

VisualArtifactSpec 1.0 (VAS)

by Shuyi Ouyang, Xiaojun Tang, Zijian Zou et al. (Accio-Lab)

What is VisualArtifactSpec 1.0 (VAS)? VisualArtifactSpec 1.0 (VAS) is an image & vision eval made by Shuyi Ouyang, Xiaojun Tang, Zijian Zou et al. (Accio-Lab). 211 creation and modification tasks in which an agent must produce a visual deliverable from a written brief and source materials: slide decks (47), static graphics such as posters and infographics (72), animations and videos (32), and web interfaces (60). Each result is scored twice: Task Fulfillment checks the stated requirements and functionality, and Artifact Performance judges content, organisation, visuals and delivery for the intended audience, using task-specific rating anchors. Prompts, rubrics and attachments are on Hugging Face, and the GitHub repo provides a local Docker judge. The paper (arXiv:2611.21382) evaluates nine model-harness setups, with overall scores from 42.1 to 70.5.

Page last updated 2026-10-18 · Evals Review

Editor note

211 creation and modification tasks in which an agent must produce a visual deliverable from a written brief and source materials: slide decks (47), static graphics such as posters and infographics (72), animations and videos (32), and web interfaces (60). Each result is scored twice: Task Fulfillment checks the stated requirements and functionality, and Artifact Performance judges content, organisation, visuals and delivery for the intended audience, using task-specific rating anchors. Prompts, rubrics and attachments are on Hugging Face, and the GitHub repo provides a local Docker judge. The paper (arXiv:2611.21382) evaluates nine model-harness setups, with overall scores from 42.1 to 70.5.

What it tests
How well agents author presentations, graphics, videos and web interfaces that meet a specification and suit their audience.
Original source
accio-lab.github.io/VisualArtifactSpec/
Last updated
v1.0; arXiv:2611.21382 (16 Nov 2026); repo last push 2026-11-17 (GitHub API, read 19 Nov 2026)
License
Dataset CC-BY-4.0 (Hugging Face, read 19 Nov 2026); repo has no licence file (GitHub API)

Facts come only from the linked source (checked 15–16 Oct 2026). Credit belongs to the creators. We don't score or rank evals: how we review.

Reviews

No reviews yet.

Reviews are written by people who sign in with GitHub, one per account per eval. We can't confirm that each reviewer used the eval; we remove reviews that break our rules, never just for being negative, and we never add, buy or edit reviews. Averages, when shown, come only from these reviews.

Sign in with GitHub to review

Related image & vision evals

  • MMMU — Expert-level multimodal understanding and reasoning.
  • MathVista — Mathematical reasoning over figures, plots and images.
  • MMBench — Broad perception and reasoning skills of vision-language models.
  • ChartQA — Question answering over charts.

All Image & vision evals · Comparison table

Discussion

No one has discussed VisualArtifactSpec 1.0 (VAS) yet. Have you run it? Share what it caught, what it missed or how you set it up.

Sign in with GitHub to comment

Be specific and kind. No promotion or affiliate links. Community rules · All discussions