Towards Better Semantic Understanding of Mobile Interfaces

Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, James W. Stout

TL;DR

This paper introduces a new, richly annotated dataset for understanding mobile user interfaces, focusing on icon shapes, semantics, and text label associations. They also release baseline models that leverage this data, demonstrating improved accessibility and automation capabilities for mobile devices, especially for visually impaired users.

What Problem Does This Paper Address?

Mobile devices are essential for daily life, but their accessibility remains a significant challenge, particularly for the estimated 2.2 billion people worldwide with vision impairments. While screen readers and accessibility guidelines have improved, a major hurdle is the lack of clear, semantic understanding of UI elements. Existing datasets and annotation methods often fall short, with many UI elements lacking explicit labels or having ambiguous ones. This gap hinders the development of advanced accessibility features and automation tools that could significantly improve the user experience for everyone.

Methods

The core of this work is the creation of a new dataset by augmenting the existing RICO dataset. They introduced three key annotation tasks:

  • Icon Shape: Annotating icons with one of 77 common shape classes, aiming to capture visual appearance.
  • Icon Semantics: Assigning semantic meanings to icons, recognizing that a single shape can have multiple functions (e.g., an "X" can mean "close," "delete," or "multiply"). This task aims to disambiguate icon functionality.
  • Label Association: Linking UI elements (icons, form fields, etc.) with their corresponding text labels, even when these associations aren't explicitly structured in the UI's view hierarchy.

To evaluate their dataset, they developed baseline models using both image-only and multimodal (image + view hierarchy + OCR) approaches. They experimented with object detection (OD) and bounding box classification (BB-CLS) architectures, including Faster R-CNN, CenterNet, and DETR.

Datasets

The primary dataset used is an enhanced version of the RICO dataset. The authors added approximately 500,000 human annotations across three tasks: Icon Shape, Icon Semantics, and Label Association. This new dataset significantly increases the number of annotated UI elements and the granularity of their categories compared to previous efforts.

Key Insights and Ideas

  • Human annotation is crucial: The paper highlights that human annotations significantly improve coverage and accuracy, especially for identifying icons, compared to automatically generated labels. They found that their human annotations identified 47% more icons than previous methods.
  • Icon semantics are complex: A single icon shape can have vastly different meanings depending on context. Disambiguating these semantics is vital for accurate UI understanding and automation.
  • View hierarchy is valuable: Incorporating view hierarchy (VH) attributes, alongside images and OCR, significantly boosts performance in bounding box classification tasks, suggesting that structured UI information is highly beneficial.
  • Label association is non-trivial: Automatically linking text labels to UI elements is challenging due to misalignments in view hierarchies and visual representations, requiring novel approaches beyond simple rule-based methods.

Key Results

  • The new dataset provides a substantial increase in annotated UI elements, with ~500k unique annotations.
  • For Icon Shape, the DETR model achieved the best performance with an [email protected] of 77.94%.
  • For Icon Semantics, DETR also performed best, achieving an [email protected] of 55.74%. Performance was generally weaker for semantics than shapes, attributed to the task's inherent difficulty.
  • In bounding box classification tasks, multimodal models (Image + VH + OCR) consistently outperformed image-only or image+OCR models. For Label Association, BB-CLS models using VH achieved an F-1 score of 87.23%, outperforming OD models.
  • The Label Association task showed that only 24.6% of icons, form fields, checkboxes, and radio buttons have associated text labels, underscoring the need for this annotation type.

Limitations and Future Work

  • Handling the "long tail": The authors acknowledge that it's difficult to cover all semantic classes with a fixed schema, leading to a significant number of "OTHER" labels. Future work could involve inferring labels for these less common icon classes.
  • Bridging the gap: There's still a performance gap between models that use only images and those that incorporate view hierarchies. Future research could focus on improving multimodal model integration and handling missing VH elements.
  • Improving VH quality: The paper suggests that improving the quality and completeness of view hierarchies in existing apps could further enhance model performance.
  • Multitask learning: Training single models to output multiple types of labels (shape, semantics, association) is a promising direction.

Summary updated

Abstract

Improving the accessibility and automation capabilities of mobile devices can have a significant positive impact on the daily lives of countless users. To stimulate research in this direction, we release a human-annotated dataset with approximately 500k unique annotations aimed at increasing the understanding of the functionality of UI elements. This dataset augments images and view hierarchies from RICO, a large dataset of mobile UIs, with annotations for icons based on their shapes and semantics, and associations between different elements and their corresponding text labels, resulting in a significant increase in the number of UI elements and the categories assigned to them. We also release models using image-only and multimodal inputs; we experiment with various architectures and study the benefits of using multimodal inputs on the new dataset. Our models demonstrate strong performance on an evaluation set of unseen apps, indicating their generalizability to newer screens. These models, combined with the new dataset, can enable innovative functionalities like referring to UI elements by their labels, improved coverage and better semantics for icons etc., which would go a long way in making UIs more usable for everyone.
BibTeX
@inproceedings{sunkara2022towards,
  author = {Srinivas Sunkara and Maria Wang and Lijuan Liu and Gilles Baechler and Yu-Chung Hsiao and Jindong Chen and Abhanshu Sharma and James W. Stout},
  title = {{Towards Better Semantic Understanding of Mobile Interfaces}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=uhfq6AvJZ4},
}

Loading related content...

Figure