Abstract
We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar. Specifically we learn a prior over mobile UI layouts, encoding common spatial co-occurrence relationships between different UI components. Conditioning region proposals using this prior leads to performance gains on UI layout parsing for both hand-drawn UIs and app screenshots, which we demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.
Introduction
Converting a UI sketch or a screenshot into a functional digital prototype is traditionally a manual, labor-intensive process. While computer vision has advanced significantly in object detection, UI layouts present unique challenges that standard detectors struggle to resolve. Hand-drawn sketches often contain visual ambiguity where a simple box could represent a button, a text field, or an image. Furthermore, standard detectors like Faster-RCNN often evaluate regions in isolation, ignoring the rich contextual "grammar" of UI design—such as the fact that a small icon adjacent to a text bar is likely a search trigger. Magic Layouts addresses these challenges by incorporating a learned structural prior that exploits the inherent spatial consistency of UI designs.
Methodology
The Magic Layouts framework augments standard object detectors with a structural prior based on component co-occurrence and regional context.
Structural Prior Components
- Horizontal Banding: To capture local context, the UI is divided into several horizontal bands. This allows the model to learn region-specific patterns, such as the frequent co-occurrence of status icons (time, battery) in the top band.
- Knowledge Graphs: Offline knowledge graphs are constructed for each band. Nodes represent UI classes, while edges represent the frequency of their co-occurrence within that specific region.
- Semantic Node Representation: The weights of the final classification layer are utilized as semantic embeddings for each component type. These embeddings provide a high-level description of component appearance (e.g., what defines a "button").
- Soft-Mapping and Propagation: When a region proposal is generated, the system uses a Gaussian function to "soft-map" the proposal to nearby bands. It then pulls information from the corresponding knowledge graph to refine the classification. For instance, a proposal in the top band that resembles an icon is reinforced as a toolbar element based on the graph's learned relationships.
Experimental Setup
The model was evaluated on two distinct datasets to test its versatility:
- RICO: A large-scale dataset of 66,000 mobile app screenshots from 9.3k Android apps, covering 25 UI component classes.
- DrawnUI: A dataset of 2,363 hand-drawn wireframe sketches across 21 UI classes, designed to test the model's robustness against the noise and variability of human sketching.
Key Results
The Magic Layouts approach consistently outperformed standard baselines, including Faster-RCNN, RetinaNet, and FPN, across both datasets.
On the RICO dataset, the method achieved an absolute increase of +4.8% mAP over standard Faster-RCNN. Notably, difficult classes such as "Page Indicator" saw improvements of up to +8.0%. On the DrawnUI dataset, the model showed a +3.5% AP average boost. It was particularly effective for rare classes; for example, detection of the "table" component improved from approximately 18% to over 51% AP. Qualitatively, the model successfully powered a web application capable of transforming whiteboard sketches into high-fidelity digital wireframes in real-time.
Discussion and Future Work
While the current model significantly improves UI parsing, it relies on horizontal banding, which is optimized for vertically scrolling mobile applications. This approach may require adaptation for desktop software or complex web dashboards featuring multi-column sidebars. Additionally, while the model handles spatial co-occurrence, it does not yet explicitly model deep hierarchical relationships (e.g., nesting a button within a container). Future research will explore these parent-child relationships to further refine the structural prior.
Summary updated