Abstract
The automated conversion of visual user interface designs into functional code is a significant challenge in software engineering. While Vision-Language Models (VLMs) have shown promise in general image understanding, the "screenshot-to-code" task is limited by the lack of high-quality, clean training data. We introduce WebSight, a large-scale synthetic dataset of 2 million HTML/screenshot pairs. By fine-tuning a foundational VLM on WebSight, we developed Sightseer, a model capable of generating clean HTML and Tailwind CSS from webpage images and handwritten sketches.
Introduction
Vision-Language Models (VLMs) have become adept at describing images, yet translating a UI mockup into functional code remains a hurdle. The primary bottleneck is the "data crisis": real-world HTML is often cluttered with scripts and complex dependencies that do not map cleanly to visual elements. To address this, we propose a synthetic data generation approach that prioritizes code cleanliness and visual alignment.
The WebSight Dataset
The core contribution of this work is WebSight, a dataset designed to teach VLMs the mapping between visual layouts and syntax. Unlike previous efforts that relied on noisy web-scraped data, WebSight is entirely synthetic.
Data Generation Pipeline
The pipeline consists of four stages:
- Concept Generation: Mistral-7B-Instruct generates millions of diverse website concepts.
- Code Generation: Deepseek-Coder-33b translates these concepts into self-contained HTML using Tailwind CSS.
- Dynamic Assets: Relevant images are integrated via the Unsplash API to ensure visual realism.
- Rendering: Playwright renders the code into high-resolution screenshots.
The full release, WebSight-v0.2, contains 2 million pairs. The use of Tailwind CSS is a strategic choice; its utility-first approach embeds styling logic directly within HTML tags, facilitating the model's ability to associate visual properties with specific code strings.
Sightseer: A Screenshot-to-Code Model
We developed Sightseer by fine-tuning a foundational VLM (Mistral-7B LLM and SigLIP-SO400M vision encoder) on the WebSight dataset. We utilized DoRA (Weight-Decomposed Low-Rank Adaptation) for parameter-efficient fine-tuning.
Key Insights and Results
Our experiments yielded several critical insights:
- Synthetic Data Superiority: Models trained on clean, synthetic data outperform those trained on "real-world" code, as the latter contains excessive noise that obscures the visual-to-code mapping.
- Zero-Shot Generalization: Despite being trained on rendered screenshots, Sightseer generalizes to handwritten sketches, enabling rapid prototyping from whiteboard drawings.
- Layout Fidelity: Sightseer demonstrates high proficiency in replicating complex layouts and preserving textual content while maintaining the "vibe" of the input image through appropriate asset selection.
Limitations and Future Directions
While Sightseer represents a significant step forward, it faces limitations in handling extremely high-density text or highly complex layouts. Occasional syntax hallucinations in Tailwind CSS also occur. Future work will explore incorporating text-only HTML/Tailwind data during pre-training to refine the model's grasp of CSS nuances.
Conclusion
By open-sourcing WebSight, we provide a foundational resource for the community to advance AI-powered design tools. The success of Sightseer underscores the power of high-quality synthetic data in bridging the gap between visual design and functional development.
Summary updated