TL;DR
VisualArtifactSpec 1.0 (VAS) is a new benchmark designed to evaluate how well AI agents create and modify complex visual outputs like presentations, infographics, and web interfaces. Unlike previous benchmarks that focus on simple instruction following, VAS introduces a dual-scoring system that separates basic task fulfillment from the nuanced quality of "artifact performance" tailored to specific audiences and media.
What Problem Does This Paper Address?
As AI agents move beyond text to generate slides, videos, and websites, we need better ways to measure their success. Current benchmarks often suffer from two main issues:
- Medium Silos: They focus on only one type of output (e.g., just websites or just slide decks), making it hard to assess an agent's general authoring capabilities.
- Binary Evaluation: They often check if a requirement was met (e.g., "Is the chart included?") but ignore the quality of the choice (e.g., "Is this chart legible on a mobile screen?").
Authoring is inherently contextual. A technical briefing for engineers requires different information density and visual hierarchy than a promotional social media carousel, even if they use the same source data. The authors address this by creating a framework that evaluates how well an agent adapts its output to the intended audience, purpose, and medium.
Methods
The researchers developed a rigorous three-stage construction workflow—spanning source preparation, task specification, and agent trial assessment—to select 211 high-quality tasks from an initial pool of 322.
The core of the VAS methodology is its two-part evaluation rubric:
- Task Fulfillment (50 pts): A binary check of "Compliance" (did you follow the constraints?) and "Functionality" (does the web interface actually work?).
- Artifact Performance (+/- 50 pts): A graded assessment across four dimensions: Content (accuracy and fit), Organization (logic and flow), Visuals (hierarchy and legibility), and Delivery (usability and editability).
To make these subjective qualities measurable, the authors use "task-specific rating anchors." These anchors provide concrete examples of what constitutes a -3 (severely deficient) versus a +3 (exceptional) score for a specific task, grounded in the intended context.
Datasets
The paper introduces VisualArtifactSpec 1.0 (VAS), a benchmark comprising 211 tasks across four "artifact families":
- Presentations (47 tasks): Narrative flow and editable structures.
- Static Graphics (72 tasks): Posters, infographics, and social media series.
- Animations & Videos (32 tasks): Processes and visual continuity over time.
- Web Interfaces (60 tasks): Interactive tools, dashboards, and workflows.
These tasks cover eight primary application purposes, ranging from research communication to practical guidance and product promotion.
Key Insights and Ideas
- Instruction Following is the Floor, Not the Ceiling: The most significant finding is that high scores in Task Fulfillment do not guarantee high-quality artifacts. Models often follow all explicit instructions but fail on "Artifact Performance"—for example, by creating a slide that is technically correct but visually unreadable.
- Performance Drives the Gap: In their evaluation of nine different model configurations, the authors found that 73.4% of the score gap between the best and worst models was due to Artifact Performance, not basic fulfillment.
- Contextual Anchors are Essential: By using specific anchors for different audiences (e.g., "first-year students" vs. "decision-makers"), the benchmark reveals failures in "Audience Alignment" that traditional metrics miss.
Key Results
The authors evaluated nine models, including GPT-6 Astra, Claude Opus 5.5, and Gemini 4 Argon.
- Top Performers: GPT-6 Astra led the pack with an overall score of 70.52, followed closely by GPT-6.1 Sol (69.37).
- The Quality Gap: While most top models achieved near-perfect Task Fulfillment (scores ~49/50), their Performance scores varied wildly (from 0.2 to 21.0).
- Common Failures: The most frequent sources of negative ratings were typography/legibility (V3) and traceability/handoff (D3). Models frequently struggled to maintain consistent markers in animations or failed to adapt text size for mobile viewing.
Limitations and Future Work
- Judge Reliability: The benchmark currently relies on an "Agent Judge" (specifically GPT-6 Astra). While the authors provide evidence for its effectiveness, future work is needed to further validate these automated judgments against human expert panels.
- Dynamic Complexity: While the benchmark includes web interfaces, the authors suggest that future versions could explore even more complex, stateful multi-step workflows.
- Expansion of Families: The current four families are broad, but there is room to include more specialized visual artifacts like 3D models or specialized CAD diagrams.
Summary updated