WAFFLE: Multi-Modal Model for Automated Front-End Development

Nan Jiang, Shanchao Liang, Nan Jiang 0012, Shangshu Qian, Lin TanarXiv 2024 · DOI · arXiv

Abstract

Automating the transition from UI design to functional code remains a significant challenge in software engineering. This paper presents WAFFLE, a novel fine-tuning strategy designed to enhance Multi-Modal Large Language Models (MLLMs) for UI-to-HTML generation. WAFFLE addresses two primary limitations of current models: structural blindness to hierarchical code and the visual-textual gap. By implementing a structure-aware attention mechanism and contrastive learning on mutated datasets, WAFFLE significantly improves the fidelity and structural integrity of generated web pages.

Introduction

The automation of front-end development—converting visual designs into clean, executable HTML and CSS—is a long-standing goal for improving developer productivity. While recent advancements in Multi-Modal Large Language Models (MLLMs) like GPT-4o have shown promise, they often struggle with the inherent complexity of web layouts. Specifically, models frequently fail to recognize the hierarchical "parent-child-sibling" relationships of the Document Object Model (DOM) and overlook subtle visual cues that dictate layout shifts. WAFFLE (Structure-Aware Fine-Tuning for Layout Engineering) is proposed to bake domain-specific knowledge of web structures directly into the model's training process.

Methodology

The WAFFLE framework employs a two-pronged approach to bridge the gap between visual input and structured code output.

The WAFFLE training pipeline featuring mutation-based contrastive learning and structure-aware attention.

Structure-Aware Attention

Standard self-attention mechanisms allow every token to attend to every other token, which can lead to a loss of structural context in hierarchical languages like HTML. WAFFLE introduces a structure-aware mask that prunes the model's attention during decoding. This mechanism forces specific attention heads to focus on three critical DOM relationships:

  • Parent-Attention: Tokens attend to their parent element's styles to facilitate CSS inheritance.
  • Sibling-Attention: Tokens attend to preceding siblings, respecting the natural flow of elements in a layout.
  • Self-Attention: Tokens focus on their own internal content and properties.
A DOM tree representation illustrating the hierarchical relationships used to guide the structure-aware attention mechanism.

This approach mimics the mental model of a human developer, ensuring that the generated code respects the logical and spatial hierarchy of the target UI.

Contrastive Fine-Tuning with Mutants

To improve the model's sensitivity to visual details, WAFFLE utilizes a mutated dataset. By intentionally introducing errors into correct HTML/image pairs—such as altering colors or shifting margins—the model is trained using a contrastive loss function. This objective encourages the model to maximize similarity between correct pairs while distinguishing them from "mutant" versions, effectively teaching the model which specific code segments are responsible for distinct visual features.

Experimental Setup and Datasets

The effectiveness of WAFFLE was evaluated using several key datasets:

  • WebSight-v0.1: A foundational training set with over 822,000 HTML/screenshot pairs.
  • WAFFLE Mutated Dataset: A specialized set of 231,940 pairs created via realistic mutation rules to facilitate contrastive learning.
  • WebSight-Test & Design2Code: Benchmarks used to assess performance on synthetic and real-world website screenshots, respectively.

Results

WAFFLE demonstrated substantial performance gains over standard fine-tuning (FT) methods across multiple metrics:

  • HTML-Match Rate: On the WebSight-Test benchmark, WAFFLE achieved a 37.00% match rate, significantly outperforming standard FT (28.00%) and commercial models like GPT-4o (11.40%).
  • Structural Fidelity: Improvements in Component-Wise Structural Similarity (CW-SSIM) by up to 0.0982 points indicate that WAFFLE-generated pages are visually much closer to the original designs.
  • Human Preference: In qualitative evaluations, human annotators consistently preferred WAFFLE's output, particularly for complex, nested layouts.
  • Model Agnosticism: The strategy proved effective across different model architectures, including VLM-WebSight and the lightweight Moondream2.

Conclusion and Future Work

WAFFLE represents a significant step forward in automated front-end development by integrating structural awareness into the MLLM training pipeline. While challenges remain regarding textual accuracy and extremely complex CSS animations, the results indicate that structural pruning and contrastive learning are powerful tools for UI-to-code tasks. Future research will explore iterative refinement steps to allow models to self-correct layout discrepancies post-rendering.

Summary updated

Abstract

Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.
BibTeX
@misc{jiang2024waffle,
  author = {Nan Jiang and Shanchao Liang and Nan Jiang and Shangshu Qian and Lin Tan},
  title = {{WAFFLE: Multi-Modal Model for Automated Front-End Development}},
  year = {2024},
  eprint = {2410.18362},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.18362},
}
Figure