WAFFLE: Fine-tuning Multi-Modal Model for Automated Front-End Development

Shanchao Liang, Nan Jiang, Shangshu Qian, Lin TanACL 2025

Introduction

Automating front-end development—the process of converting a UI design screenshot into functional HTML and CSS code—remains a significant challenge in artificial intelligence. While Multi-Modal Large Language Models (MLLMs) have shown promise in general coding tasks, they often struggle with the specific requirements of web development, such as maintaining hierarchical integrity and capturing subtle visual nuances. WAFFLE introduces a novel fine-tuning strategy designed to bridge the gap between visual design and structured code.

The Challenge of Automated Front-End Development

Current models face two primary hurdles in UI-to-code translation:

  1. Hierarchical Complexity: Unlike standard text, HTML represents a nested tree structure (the Document Object Model or DOM). Standard attention mechanisms often treat code as a linear sequence, failing to account for how parent styles or sibling layouts dictate the final visual output.
  2. The Visual-Textual Gap: MLLMs frequently overlook minor visual details—such as a 5px margin shift or specific color hex codes—that require precise CSS adjustments.

WAFFLE addresses these issues by instilling a domain-specific intuition of how HTML structures map to visual layouts.

Methodology: Structure-Awareness and Contrastive Learning

The core of WAFFLE's innovation lies in its specialized fine-tuning pipeline, which incorporates structural constraints and visual-textual alignment.

The WAFFLE training pipeline utilizing mutated data and contrastive learning.

Structure-Aware Attention

WAFFLE replaces standard self-attention with a pruned attention mask. This mechanism forces specific attention heads to prioritize three key relationships:

  • Parent-Attention: Ensures elements correctly inherit styles from their containers.
  • Sibling-Attention: Manages relative positioning and layout flow.
  • Self-Attention: Maintains the internal consistency of the element's properties.

This approach mimics the logic used by browser rendering engines, preventing "cascading failures" where a single structural error ruins the entire page.

Contrastive Fine-Tuning with Mutations

To enhance the model's sensitivity to visual details, the researchers developed a dataset of "mutants." By applying realistic mutations—such as altering colors, sizes, or removing elements—to existing HTML/CSS pairs, they created a challenging training environment.

Example of HTML code structure and its corresponding visual render, including a mutated version where a section is removed.

Using contrastive learning, the model is trained to align these specific code modifications with their visual consequences. This process teaches the model to distinguish between highly similar UIs and identify the exact code changes responsible for visual shifts.

Evaluation and Datasets

The effectiveness of WAFFLE was evaluated using three primary datasets:

  • WebSight-v0.1 (Mutated): A training set of 57,985 groups, each containing four mutated pairs, designed for contrastive learning.
  • WebSight-Test: A benchmark of 500 synthetic webpages for controlled performance evaluation.
  • Design2Code: A benchmark of 484 real-world website screenshots to test generalization to complex, "in-the-wild" designs.

Key Results

WAFFLE-tuned models demonstrated significant performance gains over state-of-the-art baselines:

  • Superiority over GPT-4o: On the WebSight-Test benchmark, WAFFLE (using a VLM-WebSight backbone) achieved a 37.00% HTML-Match score, vastly outperforming GPT-4o’s 11.40%.
  • Structural Fidelity: The model achieved up to 9.00 percentage points higher HTML match and a 0.0982 improvement in CW-SSIM (a structural image similarity metric) compared to standard fine-tuning.
  • Human Preference: Human evaluators consistently preferred WAFFLE-generated pages, noting their superior ability to replicate complex layouts accurately.

Conclusion and Future Work

WAFFLE demonstrates that incorporating structural priors and contrastive learning on mutated data can significantly improve the performance of MLLMs in front-end development tasks. While currently tested on smaller model backbones, the methodology is designed to be model-agnostic. Future research will focus on scaling these techniques to larger architectures and expanding the model's capabilities to handle more complex CSS features, such as animations and 3D transforms.

Summary updated

BibTeX
@inproceedings{liang2025waffle,
  author = {Shanchao Liang and Nan Jiang and Shangshu Qian and Lin Tan},
  title = {{WAFFLE: Fine-tuning Multi-Modal Model for Automated Front-End Development}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure