Scene-Driven Exploration and GUI Modeling for Android Apps

Xiangyu Zhang, Lingling Fan, Sen Chen, Yucheng Su, Boyuan Li

TL;DR

This paper introduces SceneDroid, a novel approach for creating detailed GUI models of Android apps by dynamically exploring "scenes" (distinct UI states) rather than just activities. SceneDroid achieves significantly higher coverage of transitions and scenes compared to existing methods, making it a valuable tool for app development, reverse engineering, and testing.

What Problem Does This Paper Address?

Developing, understanding, and testing complex Android applications is a significant challenge due to their intricate functionalities and numerous UI pages. Existing methods for modeling app GUIs, like Activity Transition Graphs (ATGs) and Window Transition Graphs (WTGs), often suffer from low transition coverage and a coarse-grained granularity. This means they miss many possible user interactions and fail to capture the nuances of UI changes, limiting their usefulness for tasks like bug detection, reverse engineering, and ensuring app quality. The core problem is the difficulty in creating comprehensive and fine-grained GUI models that accurately reflect the dynamic nature of modern mobile apps.

Methods

SceneDroid tackles this problem with a "scene-driven exploration" approach, focusing on identifying and modeling distinct UI "scenes" rather than just the broader activities. Its core methodology involves three key strategies:

  • Smart Exploration: This is the engine of SceneDroid's dynamic analysis. It combines:
  • State Fuzzing: For UI components that don't directly trigger a new scene (like text fields or checkboxes), SceneDroid intelligently fuzzes their states to explore different execution paths and uncover hidden scenes.
  • Exhaustive Exploration: Within each identified scene, SceneDroid systematically interacts with all clickable UI components to discover new scenes and transitions.
  • Indirect Launching: A crucial innovation, this strategy helps launch activities that are difficult or impossible to trigger directly via standard Inter-Component Communication (ICC) messages. It leverages the already constructed Scene Transition Graph (SceneTG) to find "upstream callers" that can indirectly lead to the target activity.
  • Scene Identification: SceneDroid defines a "scene" as a distinct UI page triggered by interactions, with a layout different from the current one. It identifies unique scenes by creating a hierarchical identifier based on component types, IDs, and their arrangement, effectively ignoring minor visual changes that don't alter the core UI structure. This allows for fine-grained modeling.
  • Scene Transition Graph (SceneTG) Construction: The identified scenes and their transitions are then organized into a SceneTG, which visually represents the app's UI flow at a granular level. Each scene in the graph is linked to its corresponding UI screenshot and layout.

Datasets

The paper uses two main types of datasets for evaluation:

  • Self-developed Benchmark Apps: The authors created 10 Android apps to serve as a ground-truth benchmark for evaluating SceneDroid's scene identification capabilities. These apps were designed to cover various UI components and complexities, reflecting real-world application design.
  • Downloaded Apps: For broader evaluation, SceneDroid was tested on a combined dataset of 50 closed-source apps from Google Play Store and 50 open-source apps from F-Droid, totaling 100 applications.

Key Insights and Ideas

  • The Power of "Scenes" over "Activities": The paper strongly argues that focusing on granular "scenes" (distinct UI states) provides a much richer and more accurate representation of an app's GUI than traditional activity-based models. This is particularly important for apps with complex UIs where a single activity can present many different visual and functional states.
  • Indirect Launching is a Game-Changer: The ability to indirectly launch activities that are otherwise inaccessible is a significant breakthrough. This overcomes a major limitation of many existing tools that rely solely on explicit ICC messages, thereby greatly improving exploration coverage.
  • Fine-Grained Modeling for Real-World Impact: By abstracting UI pages based on their hierarchical structure and component types (while ignoring minor cosmetic changes), SceneDroid achieves a level of detail that can be crucial for tasks like identifying subtle UI updates between app versions.

Key Results

SceneDroid demonstrated significant improvements over existing state-of-the-art GUI modeling tools:

  • Scene Identification: SceneDroid accurately recognized all preset scenes in its 10 self-developed benchmark apps.
  • Overall Performance: When compared against four other tools (GoalExplorer, Gator, StoryDistiller, and ICCBot) on 100 apps, SceneDroid achieved:
  • 168.74% improvement in transition pair coverage.
    • 162.42% improvement in scene extraction.
    • On average, SceneDroid extracted 30.25 transition pairs and 22.93 scenes, significantly outperforming the other tools.
  • Contribution of Strategies: An ablation study revealed that the Indirect Launching strategy had the most significant impact, followed by Scene Identification. Indirect Launching provided an average improvement of 15.59% in activity exploration, 47.02% in scene exploration, and 35.08% in transition pair extraction.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Activity Launching Failures: Despite smart exploration, some activities may still fail to launch due to reasons like authentication requirements, inconsistent declarations, or limited interaction types. The authors suggest improving interaction types and injecting system-level events.
  • Non-Native App Support: SceneDroid, like most current tools, primarily targets native Android apps. Expanding support to HTML5, cross-platform frameworks (like Flutter, React Native), and other technologies is a key future direction.

Future work includes leveraging SceneDroid for:

  • Regression Testing: Identifying fine-grained UI changes between app versions to pinpoint modifications and facilitate targeted testing.
  • Android UI Testing: Improving existing testing tools by enhancing activity coverage and enabling the development of more precise, scene-based coverage metrics.
  • GUI Fuzzing: Providing better coverage information for advanced fuzzing techniques.

Summary updated

Abstract

Due to the competitive environment, mobile apps are usually produced under pressure with lots of complicated functionality and UI pages. Therefore, it is challenging for various roles to design, understand, test, and maintain these apps. The extracted transition graphs for apps such as ATG, WTG, and STG have a low transition coverage and coarse-grained granularity, which limits the existing methods of graphical user interface (GUI) modeling by UI exploration. To solve these problems, in this paper, we propose SceneDroid, a scene-driven exploration approach to extracting the GUI scenes dynamically by integrating a series of novel techniques including smart exploration, state fuzzing, and indirect launching strategies. We present the GUI scenes as a scene transition graph (SceneTG) to model the GUI of apps with high transition coverage and fine-grained granularity. Compared with the existing GUI modeling tools, SceneDroid has improved by 168.74% in the coverage of transition pairs and 162.42% in scene extraction. Apart from the effectiveness evaluation of SceneDroid, we also illustrate the future potential of SceneDroid as a fundamental capability to support app development, reverse engineering, and GUI rearession testing.
BibTeX
@misc{zhang2023scene,
  author = {Xiangyu Zhang and Lingling Fan and Sen Chen and Yucheng Su and Boyuan Li},
  title = {{Scene-Driven Exploration and GUI Modeling for Android Apps}},
  year = {2023},
  eprint = {2308.10228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.10228},
}

Loading related content...

Figure