Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

Hugo Laurençon, Léo Tronchon, Victor SanharXiv 2024 · DOI · arXiv

Abstract

The automated conversion of visual user interface designs into functional code is a significant challenge in software engineering. While Vision-Language Models (VLMs) have shown promise in general image understanding, the "screenshot-to-code" task is limited by the lack of high-quality, clean training data. We introduce WebSight, a large-scale synthetic dataset of 2 million HTML/screenshot pairs. By fine-tuning a foundational VLM on WebSight, we developed Sightseer, a model capable of generating clean HTML and Tailwind CSS from webpage images and handwritten sketches.

Introduction

Vision-Language Models (VLMs) have become adept at describing images, yet translating a UI mockup into functional code remains a hurdle. The primary bottleneck is the "data crisis": real-world HTML is often cluttered with scripts and complex dependencies that do not map cleanly to visual elements. To address this, we propose a synthetic data generation approach that prioritizes code cleanliness and visual alignment.

The WebSight Dataset

The core contribution of this work is WebSight, a dataset designed to teach VLMs the mapping between visual layouts and syntax. Unlike previous efforts that relied on noisy web-scraped data, WebSight is entirely synthetic.

Data Generation Pipeline

The pipeline consists of four stages:

  1. Concept Generation: Mistral-7B-Instruct generates millions of diverse website concepts.
  2. Code Generation: Deepseek-Coder-33b translates these concepts into self-contained HTML using Tailwind CSS.
  3. Dynamic Assets: Relevant images are integrated via the Unsplash API to ensure visual realism.
  4. Rendering: Playwright renders the code into high-resolution screenshots.

The full release, WebSight-v0.2, contains 2 million pairs. The use of Tailwind CSS is a strategic choice; its utility-first approach embeds styling logic directly within HTML tags, facilitating the model's ability to associate visual properties with specific code strings.

Sightseer: A Screenshot-to-Code Model

We developed Sightseer by fine-tuning a foundational VLM (Mistral-7B LLM and SigLIP-SO400M vision encoder) on the WebSight dataset. We utilized DoRA (Weight-Decomposed Low-Rank Adaptation) for parameter-efficient fine-tuning.

The Sightseer pipeline: converting an original screenshot into functional code and rendering the result.

Key Insights and Results

Our experiments yielded several critical insights:

  • Synthetic Data Superiority: Models trained on clean, synthetic data outperform those trained on "real-world" code, as the latter contains excessive noise that obscures the visual-to-code mapping.
  • Zero-Shot Generalization: Despite being trained on rendered screenshots, Sightseer generalizes to handwritten sketches, enabling rapid prototyping from whiteboard drawings.
  • Layout Fidelity: Sightseer demonstrates high proficiency in replicating complex layouts and preserving textual content while maintaining the "vibe" of the input image through appropriate asset selection.
Example of a high-fidelity website rendered from code generated by Sightseer.

Limitations and Future Directions

While Sightseer represents a significant step forward, it faces limitations in handling extremely high-density text or highly complex layouts. Occasional syntax hallucinations in Tailwind CSS also occur. Future work will explore incorporating text-only HTML/Tailwind data during pre-training to refine the model's grasp of CSS nuances.

Conclusion

By open-sourcing WebSight, we provide a foundational resource for the community to advance AI-powered design tools. The success of Sightseer underscores the power of high-quality synthetic data in bridging the gap between visual design and functional development.

Summary updated

Abstract

Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight.
BibTeX
@misc{laurencon2024unlocking,
  author = {Hugo Laurençon and Léo Tronchon and Victor Sanh},
  title = {{Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset}},
  year = {2024},
  eprint = {2403.09029},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.09029},
}
Figure