AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding

Safwat Ali Khan, Wenyu Wang, Yiran Ren, Bin Zhu, Jiangfan Shi, Alyssa McGowan, Wing Lam, Kevin Moran

TL;DR

This paper introduces AURORA, a novel system that uses automated neural screen understanding to help mobile app testing tools navigate complex "tarpit" screens. By learning common UI design patterns, AURORA can classify and intelligently bypass these challenging screens, significantly improving code coverage compared to existing methods.

What Problem Does This Paper Address?

Automated testing of mobile applications is crucial for ensuring software quality, but current Automated Input Generation (AIG) tools often struggle to achieve high code coverage. A major obstacle is the presence of "UI exploration tarpits" – complex screens that cause AIG tools to get stuck, hindering their ability to explore the app's full functionality. Existing solutions, like the VET tool, address this by simply disabling these tarpit screens, which means those parts of the app remain untested. This paper aims to solve this by developing a system that can understand and navigate these tarpit screens, rather than just avoiding them.

Methods

AURORA employs a two-pronged approach:

  • Screen Recognizer: This component uses a multi-modal deep learning model to understand UI screens. It combines visual information from screenshots with textual data extracted via OCR and UI hierarchies. The goal is to classify screens into one of several predefined "UI Design Motifs," which are common patterns found in mobile app interfaces.
  • Heuristic Navigator: Once a screen is classified, AURORA applies a set of pre-defined, flexible heuristics tailored to specific UI Design Motifs. These heuristics are essentially intelligent sequences of actions designed to navigate common tarpit types (e.g., login screens, forms, advertisements).

The system is designed to work alongside existing AIG tools. When an AIG tool appears to be stuck on a screen for a certain duration, AURORA intervenes, classifies the screen, and applies the appropriate heuristic to move past it.

Datasets

  • RICO Dataset: Used for deriving UI Design Motifs. This dataset contains over 66,000 Android app UI screenshots and associated metadata.
  • VET Dataset: A dataset of tarpit screens identified from previous work, used to analyze the relationship between UI designs and tarpits and to inform heuristic development.
  • Custom Labeled Dataset: A dataset of 1369 UI screens, manually labeled into 21 UI Design Motifs, created by the authors for training and evaluating the screen classifier.

Key Insights and Ideas

  • Finite UI Design Motifs: The core idea is that there's a limited set of common UI design patterns (motifs) that appear across many mobile apps, even with variations. Recognizing these patterns is key to understanding and navigating complex screens.
  • Semantic Understanding is Crucial: Simply generating random inputs isn't enough. AIG tools need to understand the semantics of a screen (what it's for, what elements are interactive) to navigate it effectively.
  • Heuristics for Navigation: Instead of just avoiding tarpits, specific, learned heuristics can be developed to intelligently interact with and bypass them.
  • Multi-modal Approach: Combining visual and textual information from UI screens provides a richer understanding than relying on either alone.

Key Results

  • Classifier Accuracy: AURORA's CLiP-based screen classifier achieved an accuracy of 81.4% on the custom labeled dataset.
  • Tarpit Navigation Success: AURORA demonstrated an average success rate of 88.8% in navigating identified tarpit screens.
  • Method Coverage Improvement: AURORA improved method coverage by an average of 19.6% compared to the VET approach and 11.0% compared to the APE AIG tool.
  • Heuristic Effectiveness: The heuristics were successful in changing the app screen 88.8% of the time.
  • Top Tarpit Categories: Search screens, settings screens, and viewer screens were identified as the most prevalent tarpit categories.

Limitations and Future Work

  • Android 6.0 Focus: The evaluation was conducted on Android 6.0, which might limit the generalizability to newer Android versions or specific app functionalities that rely on newer APIs.
  • Manual Effort in Motif Derivation: The initial derivation of UI Design Motifs involved manual analysis, which could introduce biases.
  • Systematic vs. Random Testing: The current work focuses on improving AIG tools that are primarily random-based. Future work could explore AURORA's impact on systematic testing approaches.
  • External Validity: The study used a specific set of apps; further validation on a broader range of applications would be beneficial.

The authors suggest exploring AURORA's effect on other types of testing techniques and applying it to a wider range of applications.

Summary updated

Abstract

Nearly a decade of research in software engineering has focused on automating mobile app testing to help engineers in overcoming the unique challenges associated with the software platform. Much of this work has come in the form of Automated Input Generation tools (AIG tools) that dynamically explore app screens. However, such tools have repeatedly been demonstrated to achieve lower-than-expected code coverage - particularly on sophisticated proprietary apps. Prior work has illustrated that a primary cause of these coverage deficiencies is related to so-called tarpits, or complex screens that are difficult to navigate. In this paper, we take a critical step toward enabling AIG tools to effectively navigate tarpits during app exploration through a new form of automated semantic screen understanding. We introduce AURORA, a technique that learns from the visual and textual patterns that exist in mobile app UIs to automatically detect common screen designs and navigate them accordingly. The key idea of AURORA is that there are a finite number of mobile app screen designs, albeit with subtle variations, such that the general patterns of different categories of UI designs can be learned. As such, AURORA employs a multi-modal, neural screen classifier that is able to recognize the most common types of UI screen designs. After recognizing a given screen, it then applies a set of flexible and generalizable heuristics to properly navigate the screen. We evaluated AURORA both on a set of 12 apps with known tarpits from prior work, and on a new set of five of the most popular apps from the Google Play store. Our results indicate that AURORA is able to effectively navigate tarpit screens, outperforming prior approaches that avoid tarpits by 19.6% in terms of method coverage. The improvements can be attributed to AURORA's UI design classification and heuristic navigation techniques.
BibTeX
@inproceedings{khan2024aurora,
  author = {Safwat Ali Khan and Wenyu Wang and Yiran Ren and Bin Zhu and Jiangfan Shi and Alyssa McGowan and Wing Lam and Kevin Moran},
  title = {{AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding}},
  booktitle = {ICST},
  year = {2024},
  url = {https://openreview.net/forum?id=qQjvfSShny},
}
Figure