VisualArtifactSpec 1.0 (VAS)
by Shuyi Ouyang, Xiaojun Tang, Zijian Zou et al. (Accio-Lab)
What is VisualArtifactSpec 1.0 (VAS)? VisualArtifactSpec 1.0 (VAS) is an image & vision eval made by Shuyi Ouyang, Xiaojun Tang, Zijian Zou et al. (Accio-Lab). 211 creation and modification tasks in which an agent must produce a visual deliverable from a written brief and source materials: slide decks (47), static graphics such as posters and infographics (72), animations and videos (32), and web interfaces (60). Each result is scored twice: Task Fulfillment checks the stated requirements and functionality, and Artifact Performance judges content, organisation, visuals and delivery for the intended audience, using task-specific rating anchors. Prompts, rubrics and attachments are on Hugging Face, and the GitHub repo provides a local Docker judge. The paper (arXiv:2611.21382) evaluates nine model-harness setups, with overall scores from 42.1 to 70.5.
Editor note
211 creation and modification tasks in which an agent must produce a visual deliverable from a written brief and source materials: slide decks (47), static graphics such as posters and infographics (72), animations and videos (32), and web interfaces (60). Each result is scored twice: Task Fulfillment checks the stated requirements and functionality, and Artifact Performance judges content, organisation, visuals and delivery for the intended audience, using task-specific rating anchors. Prompts, rubrics and attachments are on Hugging Face, and the GitHub repo provides a local Docker judge. The paper (arXiv:2611.21382) evaluates nine model-harness setups, with overall scores from 42.1 to 70.5.
- What it tests
- How well agents author presentations, graphics, videos and web interfaces that meet a specification and suit their audience.
- Original source
- accio-lab.github.io/VisualArtifactSpec/
- Last updated
- v1.0; arXiv:2611.21382 (16 Nov 2026); repo last push 2026-11-17 (GitHub API, read 19 Nov 2026)
- License
- Dataset CC-BY-4.0 (Hugging Face, read 19 Nov 2026); repo has no licence file (GitHub API)
Related image & vision evals
Discussion
No one has discussed VisualArtifactSpec 1.0 (VAS) yet. Have you run it? Share what it caught, what it missed or how you set it up.