Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios

Carlos Bernal-Cárdenas, Nathan Cooper, Madeleine Havranek, Kevin Moran, Oscar Chaparro, Denys Poshyvanyk, Andrian Marcus

TL;DR

V2S+ (Video to Scenario+) is an automated tool that transforms screen recordings of Android apps into replayable test scripts using only computer vision and deep learning. By "watching" the video to detect touch indicators and classify complex gestures (including multi-finger actions like pinch-to-zoom), it allows developers to reproduce bugs and user scenarios without needing to instrument the app or have root access.

What Problem Does This Paper Address?

Mobile developers are flooded with crowdsourced feedback, often in the form of screen recordings showing bugs or feature requests. While these videos are rich in information, manually reproducing the reported behavior is tedious, time-consuming, and error-prone.

Existing "record-and-replay" tools typically suffer from two major flaws:

  1. Instrumentation Requirements: They often require developers to modify the app's source code or have root access to the device to capture events.
  2. Framework Dependency: Many tools rely on accessing the underlying GUI metadata (like XML layouts). This fails for "hybrid" apps (which use web technologies like HTML/CSS inside a native wrapper) because the metadata doesn't always reflect what is actually happening on the screen.

V2S+ tackles these challenges by operating solely on the pixels of a video file, making it framework-agnostic and easy to use with "in-the-wild" user recordings.

Methods

The V2S+ pipeline consists of three main phases:

  1. Touch Detection: The system assumes the "Show Touches" setting is enabled (which renders a small circle on the screen during interactions). It uses a Faster R-CNN model to locate these indicators in every frame. Crucially, it employs a custom Opacity CNN (a modified AlexNet) to determine if the indicator is fully opaque (finger pressed) or fading (finger lifted), which is essential for timing the start and end of actions.
  2. Action Classification: V2S+ uses a graph-based grouping algorithm to associate touches across consecutive frames. It classifies these groups into high-level actions: Tap, Long Tap, Gesture (swipes), or Multi-Fingered Gestures.
  3. Scenario Generation: The high-level actions are translated into low-level sendevent commands. These are packaged into a script and executed on a target device using a modified version of the RERAN replay engine.

Datasets

  • Training Data: The authors used the REDRAW dataset (containing 14k+ Android screenshots) to synthetically overlay touch indicators, creating a robust training set for their CV models without manual labeling.
  • Evaluation Data: An extensive set of 243 videos depicting 4,028 GUI actions. This covered:
  • 64 popular native apps from 32 Google Play categories.
  • 14 popular hybrid apps.
  • A "Multi-Touch Study" involving apps like Google Maps and Infinite Painter to test complex gestures.

Key Insights and Ideas

  • The "Opacity" Breakthrough: Traditional image processing (like Canny edge detection) struggles with varying app backgrounds. By training a Deep Learning model specifically to recognize the transparency of the touch indicator, the authors solved the difficult problem of detecting exactly when a user's finger leaves the glass.
  • Pixel-Level Universality: Because V2S+ doesn't care about the underlying code, it performs remarkably well on hybrid apps where traditional testing tools often stumble.
  • Handling "Quick" Gestures: The authors identified that a minimum of 30 FPS is required for the video to accurately resolve fast scrolling or "flicks," highlighting a hardware-software dependency in visual bug reporting.

Key Results

  • Accuracy: The Faster R-CNN achieved ~97% precision in identifying touch locations.
  • Reproduction Rate: V2S+ successfully reproduced 90.2% of sequential actions in native apps on physical devices and 83% in hybrid apps on emulators.
  • Bug Reproduction: In a controlled study of 20 apps, the tool successfully replayed 96.67% of recorded bugs and crashes.
  • Efficiency: It takes roughly 105 minutes to fully process and generate a script for a 3-minute video. While this sounds slow, the process is entirely automated and can run in the background.
  • Developer Approval: A case study with industrial partners (including developers from Microsoft and 7-Eleven) confirmed that the tool provides accurate scripts and would be highly useful for QA and debugging workflows.

Limitations and Future Work

  • Resolution Sensitivity: Currently, the object detection models are tied to specific screen resolutions. While the training process is automated, a new model must be trained for each new device aspect ratio.
  • Emulator Lag: Performance was lower on emulators due to "jagged" input from mouse movements and processing latency, suggesting that physical device recordings are preferred.
  • Coordinate Agnosticism: The current scripts use hardcoded coordinates. Future work aims to make these "coordinate-agnostic" by mapping touches to specific UI components, allowing a recording from a phone to be replayed on a tablet.

Summary updated

Abstract

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S+, an automated approach for translating video recordings of Android app usages into replayable scenarios. V2S+ is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user gestures captured in a video, and convert these into a replayable test scenario. Given that V2S+ takes a computer vision-based approach, it is applicable to both hybrid and native Android applications. We performed an extensive evaluation of V2S+ involving 243 videos depicting 4,028 GUI-based actions collected from users exercising features and reproducing bugs from a collection of over 90 popular native and hybrid Android apps. Our results illustrate that V2S+ can accurately replay scenarios from screen recordings, and is capable of reproducing $\approx$ 90.2% of sequential actions recorded in native application scenarios on physical devices, and $\approx$ 83% of sequential actions recorded in hybrid application scenarios on emulators, both with low overhead. A case study with three industrial partners illustrates the potential usefulness of V2S+ from the viewpoint of developers.
BibTeX
@article{bernalcardenas2023translating,
  author = {Carlos Bernal-Cárdenas and Nathan Cooper and Madeleine Havranek and Kevin Moran and Oscar Chaparro and Denys Poshyvanyk and Andrian Marcus},
  title = {{Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios}},
  journal = {IEEE Trans. Software Eng.},
  year = {2023},
  url = {https://openreview.net/forum?id=I5pLxRKhAG},
}

Loading related content...

Figure