TL;DR
RefCraft is a new benchmark designed to evaluate AI agents on their ability to build webpages by selectively synthesizing design elements from multiple visual references. Unlike previous benchmarks that focus on simple text-to-code or pixel-perfect reconstruction, RefCraft reveals that even frontier models like GPT-6 Astra struggle with realistic design workflows, with the best system scoring only 52.12 out of 100.
What Problem Does This Paper Address?
In the real world, "creativity is just connecting things." When a developer builds a new site, they rarely start from a blank slate or try to clone a single page exactly. Instead, they might borrow a color palette from one site, a layout from another, and a specific navigation style from a third.
Existing benchmarks for AI web development are often "idealized." They typically fall into two camps:
- Text-to-Code: Generating a page from a purely textual description, which ignores the visual nature of design.
- Screenshot-to-Code: Reconstructing a single target image perfectly, which is a rare and overly rigid task.
The authors argue that the missing link is visual-reference-conditioned agentic generation. This requires an agent to interpret subtle user instructions (e.g., "use the palette from site A but the layout from site B") and iteratively refine its output through visual inspection—a core workflow for human developers that current AI benchmarks fail to capture.
Methods
The authors developed RefCraft, a benchmark consisting of 300 diverse tasks. The core of their approach lies in how these tasks are constructed and evaluated:
- Specification-First Query Synthesis: To avoid the "mode collapse" common when prompting LLMs (where they generate repetitive, simple instructions), the authors used a structured design matrix. This matrix defines six visual dimensions (layout, palette, typography, etc.) and five requirement types: copy, adapt, invert, specify, and ignore.
- Agentic Environment: Agents operate in a "shared harness" called OpenCode. They aren't just one-shot generators; they can render their code, take a screenshot, "inspect" the result, and "revise" it over multiple turns.
- Dual-Track Judging: Because design is both objective and subjective, RefCraft uses two types of judges:
- Programmatic Judges: For deterministic requirements like color distance, contrast, and responsive behavior.
- VLM Judges: Using Gemini 3.1 Pro to assess semantic and perceptual qualities that code alone can't verify.
- Bottleneck Scoring: The final score ($S_t$) is the minimum of three components: Keypoint completion ($B_t$), Core functionality ($C_t$), and Reference-following ($R_t$). This ensures that a model isn't rewarded for a "pretty" page that is functionally broken.
Datasets
The paper introduces the RefCraft benchmark, which includes:
- 300 Multimodal Tasks: Each task pairs a natural-language instruction with 1 to 5 reference screenshots.
- 744-Page Reference Pool: A curated set of webpages from three sources: openly licensed templates, conventionally styled synthetic pages, and "atypical" synthetic pages designed to challenge model priors (e.g., unusual color combinations).
- 12,082 Evaluation Keypoints: Decomposed requirements that allow for fine-grained, verifiable assessment.
Key Insights and Ideas
- The "Adapt" Hurdle: Models are relatively good at "copying" a style or "inverting" it (doing the opposite), but they struggle significantly with "adapting" a style to a new context. This suggests that while models can recognize patterns, they find it hard to transform them creatively.
- Functionality vs. Aesthetics: A major finding was that models often satisfy individual visual requirements but fail on "core functionality"—like a button that looks right but doesn't actually trigger the requested state change.
- The Pareto Frontier of Efficiency: More tokens do not necessarily mean better results. As seen in Figure 1(c), models like Qwen3.8-Max are highly efficient, while others like GLM-5.3 use significantly more tokens for lower overall scores.
- Visual Inspection Helps, but Isn't a Silver Bullet: While the "inspect-and-revise" loop improves scores (e.g., GPT-6 Astra improved from 47.4 to 52.1), it doesn't bridge the fundamental gap in understanding complex reference relationships.
Key Results
- GPT-6 Astra leads the pack with an overall score of 52.12, followed by Qwen3.8-Max (48.95) and Grok 4.6 (48.83).
- Reference-Following is the Bottleneck: For seven out of nine models, the reference-following score ($R$) was the lowest component, highlighting that utilizing visual examples is the hardest part of the task.
- Ablation Proof: When visual references were removed, scores plummeted (e.g., GPT-6 Astra dropped from 52.1 to 17.3), proving that the benchmark successfully tests visual-conditioned generation rather than just general coding knowledge.
- Fine-Grained Performance: Models performed best on Palette (92.4) and Layout (83.1) but struggled with Interaction (43.6) and Visual Character (58.8).
Limitations and Future Work
The authors note that while their evaluation protocol is robust, more subjective criteria like "overall visual character" still show some disagreement between human experts and VLM judges.
Future work could involve:
- Expanding the benchmark to include more complex, multi-page application logic.
- Developing agents that can better "reason" about design choices rather than just reproducing attributes.
- Improving the VLM judge's ability to catch subtle "uncanny valley" design errors that humans spot instantly.
Summary updated