RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development

Yuchi Wang, Rongyao Fang, Weihao Yu, Xilin Wei, Tiancheng Gu, Dingkang Yang, Hang Zhang, Dayiheng Liu, Hongsheng Li, Shuai BaiarXiv 2026 · arXiv

TL;DR

RefCraft is a new benchmark designed to evaluate AI agents on their ability to build webpages by selectively synthesizing design elements from multiple visual references. Unlike previous benchmarks that focus on simple text-to-code or pixel-perfect reconstruction, RefCraft reveals that even frontier models like GPT-6 Astra struggle with realistic design workflows, with the best system scoring only 52.12 out of 100.

What Problem Does This Paper Address?

In the real world, "creativity is just connecting things." When a developer builds a new site, they rarely start from a blank slate or try to clone a single page exactly. Instead, they might borrow a color palette from one site, a layout from another, and a specific navigation style from a third.

Existing benchmarks for AI web development are often "idealized." They typically fall into two camps:

  1. Text-to-Code: Generating a page from a purely textual description, which ignores the visual nature of design.
  2. Screenshot-to-Code: Reconstructing a single target image perfectly, which is a rare and overly rigid task.

The authors argue that the missing link is visual-reference-conditioned agentic generation. This requires an agent to interpret subtle user instructions (e.g., "use the palette from site A but the layout from site B") and iteratively refine its output through visual inspection—a core workflow for human developers that current AI benchmarks fail to capture.

Methods

The authors developed RefCraft, a benchmark consisting of 300 diverse tasks. The core of their approach lies in how these tasks are constructed and evaluated:

  • Specification-First Query Synthesis: To avoid the "mode collapse" common when prompting LLMs (where they generate repetitive, simple instructions), the authors used a structured design matrix. This matrix defines six visual dimensions (layout, palette, typography, etc.) and five requirement types: copy, adapt, invert, specify, and ignore.
  • Agentic Environment: Agents operate in a "shared harness" called OpenCode. They aren't just one-shot generators; they can render their code, take a screenshot, "inspect" the result, and "revise" it over multiple turns.
  • Dual-Track Judging: Because design is both objective and subjective, RefCraft uses two types of judges:
  • Programmatic Judges: For deterministic requirements like color distance, contrast, and responsive behavior.
  • VLM Judges: Using Gemini 3.1 Pro to assess semantic and perceptual qualities that code alone can't verify.
  • Bottleneck Scoring: The final score ($S_t$) is the minimum of three components: Keypoint completion ($B_t$), Core functionality ($C_t$), and Reference-following ($R_t$). This ensures that a model isn't rewarded for a "pretty" page that is functionally broken.
Figure 1: Overview of RefCraft. (a) RefCraft spans a broad spectrum of practical web development domains and features visual references that vary in intensity and source. (b) An example task in which a new webpage borrows the color palette from source webpage A and the layout from source webpage B. The agent may iteratively render, inspect, and revise the webpage. (c) Overall score versus reported token usage for nine frontier models. The dashed line denotes the Pareto frontier. The results show that the task remains far from solved.

Datasets

The paper introduces the RefCraft benchmark, which includes:

  • 300 Multimodal Tasks: Each task pairs a natural-language instruction with 1 to 5 reference screenshots.
  • 744-Page Reference Pool: A curated set of webpages from three sources: openly licensed templates, conventionally styled synthetic pages, and "atypical" synthetic pages designed to challenge model priors (e.g., unusual color combinations).
  • 12,082 Evaluation Keypoints: Decomposed requirements that allow for fine-grained, verifiable assessment.

Key Insights and Ideas

  • The "Adapt" Hurdle: Models are relatively good at "copying" a style or "inverting" it (doing the opposite), but they struggle significantly with "adapting" a style to a new context. This suggests that while models can recognize patterns, they find it hard to transform them creatively.
  • Functionality vs. Aesthetics: A major finding was that models often satisfy individual visual requirements but fail on "core functionality"—like a button that looks right but doesn't actually trigger the requested state change.
  • The Pareto Frontier of Efficiency: More tokens do not necessarily mean better results. As seen in Figure 1(c), models like Qwen3.8-Max are highly efficient, while others like GLM-5.3 use significantly more tokens for lower overall scores.
  • Visual Inspection Helps, but Isn't a Silver Bullet: While the "inspect-and-revise" loop improves scores (e.g., GPT-6 Astra improved from 47.4 to 52.1), it doesn't bridge the fundamental gap in understanding complex reference relationships.
The construction and evaluation pipeline of RefCraft, from reference pool curation to dual-track judging.

Key Results

  • GPT-6 Astra leads the pack with an overall score of 52.12, followed by Qwen3.8-Max (48.95) and Grok 4.6 (48.83).
  • Reference-Following is the Bottleneck: For seven out of nine models, the reference-following score ($R$) was the lowest component, highlighting that utilizing visual examples is the hardest part of the task.
  • Ablation Proof: When visual references were removed, scores plummeted (e.g., GPT-6 Astra dropped from 52.1 to 17.3), proving that the benchmark successfully tests visual-conditioned generation rather than just general coding knowledge.
  • Fine-Grained Performance: Models performed best on Palette (92.4) and Layout (83.1) but struggled with Interaction (43.6) and Visual Character (58.8).
Fine-grained performance breakdown by visual dimension and requirement type.

Limitations and Future Work

The authors note that while their evaluation protocol is robust, more subjective criteria like "overall visual character" still show some disagreement between human experts and VLM judges.

Future work could involve:

  • Expanding the benchmark to include more complex, multi-page application logic.
  • Developing agents that can better "reason" about design choices rather than just reproducing attributes.
  • Improving the VLM judge's ability to catch subtle "uncanny valley" design errors that humans spot instantly.

Summary updated

Abstract

Beyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks. We study this crucial capability through webpage development, a practical and readily evaluable setting in which existing benchmarks remain largely idealized: they typically focus on reconstructing visual targets or generating webpages from textual instructions alone, overlooking the more common workflow of developing from visual examples. Specifically, we introduce RefCraft, a benchmark comprising 300 realistic multimodal webpage-development tasks that combine natural-language instructions with screenshots from one to five reference webpages. Agents must selectively interpret and utilize these references while rendering, visually inspecting, and iteratively refining their outputs. We evaluate the resulting open-ended artifacts using independently verifiable keypoints and complementary programmatic and VLM judges. Across nine frontier (M)LLMs, the strongest system achieves an overall score of only 52.12 out of 100, revealing that visual-reference-conditioned agentic generation remains a substantial challenge for current models in practical webpage-development settings.
BibTeX
@misc{wang2026refcraft,
  author = {Yuchi Wang and Rongyao Fang and Weihao Yu and Xilin Wei and Tiancheng Gu and Dingkang Yang and Hang Zhang and Dayiheng Liu and Hongsheng Li and Shuai Bai},
  title = {{RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development}},
  year = {2026},
  eprint = {2610.26826},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26826},
}
Figure