Magic Layouts: Structural Prior for Component Detection in User Interface Designs

Dipu Manandhar, Hailin Jin, John P. Collomosse

Abstract

Converting hand-drawn sketches or mobile screenshots into functional digital prototypes is a labor-intensive manual process. While deep learning has advanced object detection, user interface (UI) components often lack unique visual signatures, leading to significant ambiguity. We present "Magic Layouts," a framework that incorporates a structural prior—learned spatial relationships and co-occurrence patterns—into detection models. By teaching models the "grammar" of UI design, we achieve significant performance gains in parsing both professional mobile app screenshots and messy, hand-drawn wireframes.

Introduction

Standard object detectors, such as Faster-RCNN, typically operate as "local" thinkers, classifying regions based solely on visual features. In the domain of UI design, this approach is insufficient. A simple rectangle may represent a button, an icon, or a checkbox depending entirely on its surrounding context. For instance, a rectangle at the top of a screen adjacent to a "hamburger" icon is likely a search bar.

Existing models often lack this global structural context. They do not inherently understand that certain elements, like text input fields and submit buttons, are frequent neighbors, or that toolbars are almost exclusively located at the top of a mobile interface. Magic Layouts addresses this gap by encoding these design rules directly into the detection process.

Methodology: The Structural Prior

The core innovation of this work is the Object Co-occurrence Knowledge Graph, which provides the model with a learned understanding of UI layouts through several key steps:

  1. Horizontal Banding: The framework divides a UI layout into several horizontal bands, recognizing that UI statistics vary significantly along the vertical axis.
  2. Knowledge Graph Construction: By analyzing thousands of existing layouts, the system constructs a Knowledge Graph for each band. These graphs encode the frequency with which different components appear together in specific regions.
  3. Semantic Node Representation: Rather than relying on complex visual embeddings, the model utilizes the weights of its own classifier as "semantic nodes," allowing it to propagate information through the graph using high-level categorical understanding.
The Magic Layouts architecture integrates knowledge graphs and propagation to condition proposal features with structural context.

When identifying a potential object (a region proposal), the model does not rely on visual data alone. It consults the Knowledge Graphs of nearby bands and uses a Gaussian function to "soft-map" the proposal to these graphs, combining raw visual features with the learned structural prior to reach a final classification.

Evaluation and Results

The researchers evaluated Magic Layouts on two primary benchmarks: the RICO Dataset (66,000 professional mobile app screenshots) and the DrawnUI Dataset (2,363 hand-drawn sketches).

Quantitative Performance

The results demonstrate a consistent "Magic" boost across various state-of-the-art backbones, including Faster-RCNN, DC5, and Feature Pyramid Networks (FPN). On the RICO dataset, the method achieved an absolute increase of +4.8% mAP over the Faster-RCNN baseline. On the more challenging DrawnUI dataset, the FPN+Magic configuration reached 64.3% mAP.

Average Precision (AP) comparison across various UI components shows consistent improvements when the "Magic" structural prior is applied.

Key Insights

  • Location-Awareness: UI design follows a predictable "grammar." Leveraging region-specific expectations helps resolve visual ambiguity.
  • Boosting Rare Classes: The model significantly improves detection for rare components. In the DrawnUI dataset, Average Precision for "tables" jumped from ~18% to over 51% by leveraging the structural context in which they typically appear.
  • Architecture Agnostic: The framework is a modular enhancement that improves various existing detection backbones.

Conclusion and Future Work

Magic Layouts demonstrates that incorporating structural priors can significantly enhance the accuracy of automated UI parsing. While the current horizontal banding approach is highly effective for mobile UIs, future research may explore more complex, nested graph structures to capture hierarchical "parent-child" relationships. This would extend the framework's utility to more complex desktop and multi-column web layouts.

Summary updated

Abstract

We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar. Specifically we learn a prior over mobile UI layouts, encoding common spatial co-occurrence relationships between different UI components. Conditioning region proposals using this prior leads to performance gains on UI layout parsing for both hand-drawn UIs and app screenshots, which we demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.
BibTeX
@inproceedings{manandhar2021magic,
  author = {Dipu Manandhar and Hailin Jin and John P. Collomosse},
  title = {{Magic Layouts: Structural Prior for Component Detection in User Interface Designs}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=5RJaLrMjyCUQ},
}
Figure