VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications

Shuyi Ouyang, Xiaojun Tang, Zijian Zou, Xintong Zhang, Mingli Song, Guannan Zhang, Hongwei XuearXiv 2026 · arXiv

TL;DR

VisualArtifactSpec 1.0 (VAS) is a new benchmark designed to evaluate how well AI agents create and modify complex visual outputs like presentations, infographics, and web interfaces. Unlike previous benchmarks that focus on simple instruction following, VAS introduces a dual-scoring system that separates basic task fulfillment from the nuanced quality of "artifact performance" tailored to specific audiences and media.

What Problem Does This Paper Address?

As AI agents move beyond text to generate slides, videos, and websites, we need better ways to measure their success. Current benchmarks often suffer from two main issues:

  1. Medium Silos: They focus on only one type of output (e.g., just websites or just slide decks), making it hard to assess an agent's general authoring capabilities.
  2. Binary Evaluation: They often check if a requirement was met (e.g., "Is the chart included?") but ignore the quality of the choice (e.g., "Is this chart legible on a mobile screen?").

Authoring is inherently contextual. A technical briefing for engineers requires different information density and visual hierarchy than a promotional social media carousel, even if they use the same source data. The authors address this by creating a framework that evaluates how well an agent adapts its output to the intended audience, purpose, and medium.

Figure 2 illustrates how the same source material should be organized and presented differently depending on the audience and purpose.

Methods

The researchers developed a rigorous three-stage construction workflow—spanning source preparation, task specification, and agent trial assessment—to select 211 high-quality tasks from an initial pool of 322.

The three-stage construction workflow ensures tasks are feasible, verifiable, and diverse.

The core of the VAS methodology is its two-part evaluation rubric:

  • Task Fulfillment (50 pts): A binary check of "Compliance" (did you follow the constraints?) and "Functionality" (does the web interface actually work?).
  • Artifact Performance (+/- 50 pts): A graded assessment across four dimensions: Content (accuracy and fit), Organization (logic and flow), Visuals (hierarchy and legibility), and Delivery (usability and editability).

To make these subjective qualities measurable, the authors use "task-specific rating anchors." These anchors provide concrete examples of what constitutes a -3 (severely deficient) versus a +3 (exceptional) score for a specific task, grounded in the intended context.

The evaluation framework separates basic fulfillment from high-level performance.

Datasets

The paper introduces VisualArtifactSpec 1.0 (VAS), a benchmark comprising 211 tasks across four "artifact families":

  • Presentations (47 tasks): Narrative flow and editable structures.
  • Static Graphics (72 tasks): Posters, infographics, and social media series.
  • Animations & Videos (32 tasks): Processes and visual continuity over time.
  • Web Interfaces (60 tasks): Interactive tools, dashboards, and workflows.
The 211 tasks are distributed across four families and 14 specific task types.

These tasks cover eight primary application purposes, ranging from research communication to practical guidance and product promotion.

Tasks are categorized by their real-world application context.

Key Insights and Ideas

  • Instruction Following is the Floor, Not the Ceiling: The most significant finding is that high scores in Task Fulfillment do not guarantee high-quality artifacts. Models often follow all explicit instructions but fail on "Artifact Performance"—for example, by creating a slide that is technically correct but visually unreadable.
  • Performance Drives the Gap: In their evaluation of nine different model configurations, the authors found that 73.4% of the score gap between the best and worst models was due to Artifact Performance, not basic fulfillment.
  • Contextual Anchors are Essential: By using specific anchors for different audiences (e.g., "first-year students" vs. "decision-makers"), the benchmark reveals failures in "Audience Alignment" that traditional metrics miss.

Key Results

The authors evaluated nine models, including GPT-6 Astra, Claude Opus 5.5, and Gemini 4 Argon.

  • Top Performers: GPT-6 Astra led the pack with an overall score of 70.52, followed closely by GPT-6.1 Sol (69.37).
  • The Quality Gap: While most top models achieved near-perfect Task Fulfillment (scores ~49/50), their Performance scores varied wildly (from 0.2 to 21.0).
  • Common Failures: The most frequent sources of negative ratings were typography/legibility (V3) and traceability/handoff (D3). Models frequently struggled to maintain consistent markers in animations or failed to adapt text size for mobile viewing.

Limitations and Future Work

  • Judge Reliability: The benchmark currently relies on an "Agent Judge" (specifically GPT-6 Astra). While the authors provide evidence for its effectiveness, future work is needed to further validate these automated judgments against human expert panels.
  • Dynamic Complexity: While the benchmark includes web interfaces, the authors suggest that future versions could explore even more complex, stateful multi-step workflows.
  • Expansion of Families: The current four families are broad, but there is room to include more specialized visual artifacts like 3D models or specialized CAD diagrams.

Summary updated

Abstract

Visual artifact tasks test agents' ability to translate task specifications and source materials into outputs that meet practical needs. Evaluation across artifact families helps distinguish broadly applicable authoring capabilities from strengths confined to a particular medium. We introduce VisualArtifactSpec 1.0 (VAS), a benchmark of 211 creation and modification tasks spanning presentations, static graphics, animations and videos, and web interfaces. Each task specifies authoring objectives, source materials, constraints, and deliverables. Existing benchmarks provide limited support for systematically evaluating how authoring choices suit different audiences, purposes, and media across artifact families. VAS evaluates two aspects separately: Task Fulfillment verifies explicit requirements and applicable functionality, while Artifact Performance assesses Content, Organization, Visuals, and Delivery. Task-specific rating anchors instantiate shared criteria for the intended audience, purpose, and medium, with judgments grounded in source materials and artifact inspection records. We evaluate nine model-harness configurations on all 211 tasks. Overall scores range from 42.12 to 70.52, with Artifact Performance accounting for 73.4% of the gap between the highest- and lowest-scoring configurations.
BibTeX
@misc{ouyang2026visualartifactspec,
  author = {Shuyi Ouyang and Xiaojun Tang and Zijian Zou and Xintong Zhang and Mingli Song and Guannan Zhang and Hongwei Xue},
  title = {{VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications}},
  year = {2026},
  eprint = {2611.21382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21382},
}
Figure