Abstract
Automating the transition from UI design to functional code remains a significant challenge in software engineering. This paper presents WAFFLE, a novel fine-tuning strategy designed to enhance Multi-Modal Large Language Models (MLLMs) for UI-to-HTML generation. WAFFLE addresses two primary limitations of current models: structural blindness to hierarchical code and the visual-textual gap. By implementing a structure-aware attention mechanism and contrastive learning on mutated datasets, WAFFLE significantly improves the fidelity and structural integrity of generated web pages.
Introduction
The automation of front-end development—converting visual designs into clean, executable HTML and CSS—is a long-standing goal for improving developer productivity. While recent advancements in Multi-Modal Large Language Models (MLLMs) like GPT-4o have shown promise, they often struggle with the inherent complexity of web layouts. Specifically, models frequently fail to recognize the hierarchical "parent-child-sibling" relationships of the Document Object Model (DOM) and overlook subtle visual cues that dictate layout shifts. WAFFLE (Structure-Aware Fine-Tuning for Layout Engineering) is proposed to bake domain-specific knowledge of web structures directly into the model's training process.
Methodology
The WAFFLE framework employs a two-pronged approach to bridge the gap between visual input and structured code output.
Structure-Aware Attention
Standard self-attention mechanisms allow every token to attend to every other token, which can lead to a loss of structural context in hierarchical languages like HTML. WAFFLE introduces a structure-aware mask that prunes the model's attention during decoding. This mechanism forces specific attention heads to focus on three critical DOM relationships:
- Parent-Attention: Tokens attend to their parent element's styles to facilitate CSS inheritance.
- Sibling-Attention: Tokens attend to preceding siblings, respecting the natural flow of elements in a layout.
- Self-Attention: Tokens focus on their own internal content and properties.
This approach mimics the mental model of a human developer, ensuring that the generated code respects the logical and spatial hierarchy of the target UI.
Contrastive Fine-Tuning with Mutants
To improve the model's sensitivity to visual details, WAFFLE utilizes a mutated dataset. By intentionally introducing errors into correct HTML/image pairs—such as altering colors or shifting margins—the model is trained using a contrastive loss function. This objective encourages the model to maximize similarity between correct pairs while distinguishing them from "mutant" versions, effectively teaching the model which specific code segments are responsible for distinct visual features.
Experimental Setup and Datasets
The effectiveness of WAFFLE was evaluated using several key datasets:
- WebSight-v0.1: A foundational training set with over 822,000 HTML/screenshot pairs.
- WAFFLE Mutated Dataset: A specialized set of 231,940 pairs created via realistic mutation rules to facilitate contrastive learning.
- WebSight-Test & Design2Code: Benchmarks used to assess performance on synthetic and real-world website screenshots, respectively.
Results
WAFFLE demonstrated substantial performance gains over standard fine-tuning (FT) methods across multiple metrics:
- HTML-Match Rate: On the WebSight-Test benchmark, WAFFLE achieved a 37.00% match rate, significantly outperforming standard FT (28.00%) and commercial models like GPT-4o (11.40%).
- Structural Fidelity: Improvements in Component-Wise Structural Similarity (CW-SSIM) by up to 0.0982 points indicate that WAFFLE-generated pages are visually much closer to the original designs.
- Human Preference: In qualitative evaluations, human annotators consistently preferred WAFFLE's output, particularly for complex, nested layouts.
- Model Agnosticism: The strategy proved effective across different model architectures, including VLM-WebSight and the lightweight Moondream2.
Conclusion and Future Work
WAFFLE represents a significant step forward in automated front-end development by integrating structural awareness into the MLLM training pipeline. While challenges remain regarding textual accuracy and extremely complex CSS animations, the results indicate that structural pruning and contrastive learning are powerful tools for UI-to-code tasks. Future research will explore iterative refinement steps to allow models to self-correct layout discrepancies post-rendering.
Summary updated