Owl Eyes: Spotting UI Display Issues via Visual Understanding

Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, Qing Wang

Abstract

Ensuring a consistent User Interface (UI) across the fragmented Android ecosystem is a significant challenge for developers. Traditional automated testing tools often fail to detect "silent" visual bugs like overlapping text or missing images. This paper introduces OwlEye, a deep-learning-based framework that treats UI testing as a computer vision problem. By utilizing a Convolutional Neural Network (CNN) and a novel heuristic-based data augmentation strategy, OwlEye automatically detects and localizes visual glitches with high precision.

Introduction

The Android ecosystem's diversity, with over 24,000 device models and numerous OS versions, makes manual UI verification costly and reactive. While automated tools like "Monkey" are effective at detecting crashes, they are blind to display issues—glitches like text overlapping, components being occluded, or images failing to load—that degrade user experience without stopping the application. OwlEye addresses this gap by "looking" at screenshots like a human tester to identify defects before an app is released.

Examples of common UI display issues detected by OwlEye, including overlapping text and missing images.

Methodology

OwlEye's architecture is built on three main pillars:

  1. CNN-based Detection: A 12-layer Convolutional Neural Network performs binary classification on UI screenshots to identify "buggy" vs. "normal" states.
  2. Grad-CAM Localization: To assist developers in fixing bugs, the system employs Gradient-weighted Class Activation Mapping (Grad-CAM). This technique generates heatmaps over screenshots, highlighting the specific regions that triggered the "buggy" classification.
  3. Heuristic-based Data Augmentation: To overcome the scarcity of labeled UI bug data, the researchers developed an automated generator. This tool takes clean screenshots and programmatically injects common issues, such as overlapping text fields or replacing images with "missing" icons, to create a robust training set.

Experimental Setup and Datasets

The model was trained and evaluated using two primary data sources:

  • Crowdtesting Dataset: 10,330 unique screenshots collected from Baidu, with 4,470 manually labeled as containing display issues.
  • Rico Dataset: A publicly available dataset of over 66,000 UI screenshots used as a base for synthetic bug generation.
  • Benchmark: A balanced dataset of 15,600 screenshots (half augmented buggy, half clean) was released for follow-up research.

Key Results and Insights

OwlEye achieved 85% precision and 84% recall in detecting UI display issues, significantly outperforming traditional computer vision baselines like SIFT, SURF, and ORB. In a real-world evaluation of 329 popular apps from Google Play and F-Droid, OwlEye uncovered 57 previously undetected UI issues, 26 of which were confirmed or fixed by developers at the time of publication.

A key insight is the effectiveness of visual understanding over programmatic analysis. By treating the UI as pixels rather than code, OwlEye remains platform-agnostic. Furthermore, the integration of Grad-CAM provides necessary explainability, reducing the manual effort required for debugging.

Limitations and Future Work

Despite its success, OwlEye faces challenges with extremely small UI defects that occupy a tiny fraction of the screen. Additionally, the "blurred screen" category remains difficult to detect and simulate due to its rarity in the training data.

An example of a "blurred screen" or corrupted UI rendering, which represents a challenging category for automated detection.

Future research will focus on linking visual heatmaps back to the underlying view hierarchy to provide root cause analysis, helping developers pinpoint the exact code attribute causing the glitch.

Summary updated

BibTeX
@inproceedings{liu2020owl,
  author = {Zhe Liu and Chunyang Chen and Junjie Wang and Yuekai Huang and Jun Hu and Qing Wang},
  title = {{Owl Eyes: Spotting UI Display Issues via Visual Understanding}},
  booktitle = {ASE},
  year = {2020},
  doi = {10.1145/3324884.3416547},
}
Figure