TL;DR
This paper introduces Mobile-UI-Repair (M-UI-R), a novel deep learning approach that uses Faster R-CNN to automatically detect and locate UI display bugs in mobile applications from screenshots. This method significantly outperforms existing techniques, offering a more efficient and accurate way for developers to identify and fix visual issues that impact user experience.
What Problem Does This Paper Address?
Mobile applications are ubiquitous, and their user interface (UI) is critical for usability and user satisfaction. However, developing these apps, especially across the diverse landscape of Android devices with varying screen resolutions and OS versions, often leads to UI display issues. These "UI smells" or bugs, such as text overlap, missing images, and component occlusion, can degrade the user experience, even if they don't cause app crashes. Manual testing to find these issues is time-consuming, inefficient, and prone to missing subtle problems. Existing automated testing methods often struggle with these visual defects, focusing more on functional bugs. This paper aims to bridge this gap by providing an automated, visual-based solution for detecting and localizing these UI display bugs.
Methods
The core of M-UI-R is the Faster R-CNN deep learning model, a well-established framework for object detection. The authors frame UI display bug detection as an object detection task, where the "objects" are the problematic UI elements or regions.
Here's a breakdown of the approach:
- Feature Extraction: The model uses ResNet50, a deep convolutional neural network known for its effectiveness in image classification, to extract rich visual features from UI screenshots. ResNet50's residual learning architecture helps overcome the degradation issues common in very deep networks.
- Region Proposal Network (RPN): This component takes the feature maps from ResNet50 and generates "region proposals" – potential bounding boxes that might contain UI bugs. It uses a sliding window and anchors of various sizes and ratios to identify these regions.
- Region of Interest (ROI) Pooling: This module extracts fixed-size feature maps from the proposals generated by the RPN.
- Classification and Localization: Finally, fully connected layers with a softmax activation are used to classify these features into specific UI bug categories (component occlusion, missing images, text overlap, null values) and refine the bounding boxes for precise localization of the identified bugs.
This approach leverages visual information directly from screenshots, making it adaptable across different platforms and less reliant on syntactical code analysis.
Datasets
The study utilized a combination of datasets:
- Rico Dataset: This publicly available dataset was used for training and testing. It's a large collection of UI screenshots from mobile applications.
- Crowd-sourcing Platform (Baidu): Data was collected from users testing mobile applications, providing screenshots and bug reports. This dataset included 1,443 testing tasks for Android apps.
- Software Houses: UI screenshots from applications with over 5,000 downloads and high ratings (e.g., StarzPlay, Perfect Piano) were collected.
In total, 32,000 UI screenshots were used for analysis, with 8,000 from each of the four bug categories (text overlap, component occlusion, missing images, null values). Half of these were positive (buggy) and half negative (bug-free). The data was split with 80% for training and 20% for testing and validation. A separate real-time dataset of 800 screenshots (400 buggy, 400 bug-free) from crowd-testing apps was also used for validation.
Key Insights and Ideas
- Visualizing Bugs is Key: The paper strongly advocates for a visual understanding approach, mimicking human perception to identify UI display issues, which is more effective than solely relying on static code analysis or textual bug reports for these specific problems.
- Faster R-CNN's Suitability: Framing UI bug detection as an object detection problem and applying Faster R-CNN proved highly effective, allowing for both identification and precise localization of bugs within screenshots.
- Beyond Functional Bugs: M-UI-R specifically targets non-functional, visual UI bugs that can significantly impact usability without necessarily causing app crashes, a critical area often overlooked by traditional testing.
- Developer Validation: The survey with developers confirmed that the information provided by M-UI-R is valuable and helps them understand the impact of UI bugs, validating the practical utility of the approach.
Key Results
- Detection Performance: M-UI-R achieved an average precision of 87.7% and an average recall of 86.5% on the training dataset for UI display issue detection. On real-world data, it achieved 85.6% precision and 84.3% recall.
- Localization Performance: For UI bug localization, M-UI-R achieved an average precision of 71.5% and an average recall of 69.7% on real and testing datasets.
- Outperformance: M-UI-R significantly outperformed baseline methods, including machine learning approaches and the OwlEye deep learning model. For instance, it showed a 28.8% higher precision and 21.7% higher recall compared to the multilayer perceptron (MLP) baseline. It also detected 79 additional issues that OwlEye missed.
- Developer Feedback: Seven out of eight developers found M-UI-R effective for detecting UI display issues, and six strongly agreed on its usefulness. Six out of eight also agreed on its effectiveness for localization.
Limitations and Future Work
- Generalizability to Other Platforms: While the study focused on mobile apps, the authors believe the approach can be extended to other platforms (iOS, Mac OS, Windows) and languages, with preliminary experiments showing promise.
- Design Smell Scope: M-UI-R currently focuses on specific visual bugs (text overlap, occlusion, missing images, null values) and does not identify design smells related to UI style, layout, typography, or iconography.
- Parameter Sensitivity: Classification algorithms can be sensitive to hyperparameter values, and while the authors experimented with variations, these adjustments could still impact results.
- Object Detection Model Choice: The use of Faster R-CNN was based on its superior performance at the time, but the absence of more advanced object detection tools for UI bug detection could be a limitation.
- Future Work: The authors plan to improve classification performance, delve into pinpointing the underlying causes of bugs, and develop tools to recommend patches to developers. They also aim to incorporate attention mechanisms and image magnification for better detection of subtle issues.
Summary updated