TL;DR
This paper introduces AURORA, a novel system that uses automated neural screen understanding to help mobile app testing tools navigate complex "tarpit" screens. By learning common UI design patterns, AURORA can classify and intelligently bypass these challenging screens, significantly improving code coverage compared to existing methods.
What Problem Does This Paper Address?
Automated testing of mobile applications is crucial for ensuring software quality, but current Automated Input Generation (AIG) tools often struggle to achieve high code coverage. A major obstacle is the presence of "UI exploration tarpits" – complex screens that cause AIG tools to get stuck, hindering their ability to explore the app's full functionality. Existing solutions, like the VET tool, address this by simply disabling these tarpit screens, which means those parts of the app remain untested. This paper aims to solve this by developing a system that can understand and navigate these tarpit screens, rather than just avoiding them.
Methods
AURORA employs a two-pronged approach:
- Screen Recognizer: This component uses a multi-modal deep learning model to understand UI screens. It combines visual information from screenshots with textual data extracted via OCR and UI hierarchies. The goal is to classify screens into one of several predefined "UI Design Motifs," which are common patterns found in mobile app interfaces.
- Heuristic Navigator: Once a screen is classified, AURORA applies a set of pre-defined, flexible heuristics tailored to specific UI Design Motifs. These heuristics are essentially intelligent sequences of actions designed to navigate common tarpit types (e.g., login screens, forms, advertisements).
The system is designed to work alongside existing AIG tools. When an AIG tool appears to be stuck on a screen for a certain duration, AURORA intervenes, classifies the screen, and applies the appropriate heuristic to move past it.
Datasets
- RICO Dataset: Used for deriving UI Design Motifs. This dataset contains over 66,000 Android app UI screenshots and associated metadata.
- VET Dataset: A dataset of tarpit screens identified from previous work, used to analyze the relationship between UI designs and tarpits and to inform heuristic development.
- Custom Labeled Dataset: A dataset of 1369 UI screens, manually labeled into 21 UI Design Motifs, created by the authors for training and evaluating the screen classifier.
Key Insights and Ideas
- Finite UI Design Motifs: The core idea is that there's a limited set of common UI design patterns (motifs) that appear across many mobile apps, even with variations. Recognizing these patterns is key to understanding and navigating complex screens.
- Semantic Understanding is Crucial: Simply generating random inputs isn't enough. AIG tools need to understand the semantics of a screen (what it's for, what elements are interactive) to navigate it effectively.
- Heuristics for Navigation: Instead of just avoiding tarpits, specific, learned heuristics can be developed to intelligently interact with and bypass them.
- Multi-modal Approach: Combining visual and textual information from UI screens provides a richer understanding than relying on either alone.
Key Results
- Classifier Accuracy: AURORA's CLiP-based screen classifier achieved an accuracy of 81.4% on the custom labeled dataset.
- Tarpit Navigation Success: AURORA demonstrated an average success rate of 88.8% in navigating identified tarpit screens.
- Method Coverage Improvement: AURORA improved method coverage by an average of 19.6% compared to the VET approach and 11.0% compared to the APE AIG tool.
- Heuristic Effectiveness: The heuristics were successful in changing the app screen 88.8% of the time.
- Top Tarpit Categories: Search screens, settings screens, and viewer screens were identified as the most prevalent tarpit categories.
Limitations and Future Work
- Android 6.0 Focus: The evaluation was conducted on Android 6.0, which might limit the generalizability to newer Android versions or specific app functionalities that rely on newer APIs.
- Manual Effort in Motif Derivation: The initial derivation of UI Design Motifs involved manual analysis, which could introduce biases.
- Systematic vs. Random Testing: The current work focuses on improving AIG tools that are primarily random-based. Future work could explore AURORA's impact on systematic testing approaches.
- External Validity: The study used a specific set of apps; further validation on a broader range of applications would be beneficial.
The authors suggest exploring AURORA's effect on other types of testing techniques and applying it to a wider range of applications.
Summary updated