Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios

Carlos Bernal-Cárdenas, Nathan Cooper, Madeleine Havranek, Kevin Moran, Oscar Chaparro, Denys Poshyvanyk, Andrian Marcus

Abstract

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S+, an automated approach for translating video recordings of Android app usages into replayable scenarios. V2S+ is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user gestures captured in a video, and convert these into a replayable test scenario. Given that V2S+ takes a computer vision-based approach, it is applicable to both hybrid and native Android applications. We performed an extensive evaluation of V2S+ involving 243 videos depicting 4,028 GUI-based actions collected from users exercising features and reproducing bugs from a collection of over 90 popular native and hybrid Android apps. Our results illustrate that V2S+ can accurately replay scenarios from screen recordings, and is capable of reproducing $\approx$ 90.2% of sequential actions recorded in native application scenarios on physical devices, and $\approx$ 83% of sequential actions recorded in hybrid application scenarios on emulators, both with low overhead. A case study with three industrial partners illustrates the potential usefulness of V2S+ from the viewpoint of developers.

Introduction

Mobile developers are increasingly reliant on crowdsourced feedback to identify bugs and refine user experiences. Screen recordings have emerged as a preferred medium for this feedback because they capture the dynamic context of an app's usage. However, the transition from a video file to a reproducible bug report is fraught with manual effort. Developers must watch the video, identify the sequence of interactions, and manually attempt to replicate them on a test device—a process that is time-consuming and prone to human error.

Existing "record-and-replay" solutions attempt to automate this but often require intrusive instrumentation of the application's source code or root access to the device. Furthermore, many tools rely on underlying GUI metadata (such as XML layouts), which frequently fails to represent the actual visual state in "hybrid" applications—those built using web technologies like HTML and CSS wrapped in a native container. To bridge this gap, we present V2S+ (Video to Scenario+), a framework-agnostic tool that translates screen recordings into replayable scripts using only computer vision and deep learning.

The V2S+ Approach

V2S+ operates solely on the pixel data of a video file, making it applicable to any Android application regardless of its underlying framework. The pipeline is divided into three distinct phases:

1. Touch Detection

The system leverages the "Show Touches" developer setting in Android, which renders a visual indicator (a small circle) whenever a user interacts with the screen. V2S+ employs a Faster R-CNN model to locate these indicators across every frame of the video. To solve the challenge of timing—knowing exactly when a finger touches or leaves the glass—the authors developed a custom Opacity CNN (based on AlexNet). This model analyzes the transparency of the touch indicator; because the Android OS fades the indicator upon release, the CNN can distinguish between a fully opaque active touch and a fading inactive one with high precision.

2. Action Classification

Once touch locations are identified, V2S+ uses a graph-based grouping algorithm to track touches across consecutive frames. These groups are then classified into high-level user actions, including:

  • Taps and Long Taps
  • Gestures (Swipes/Flicks)
  • Multi-Finger Gestures (e.g., pinch-to-zoom)

3. Scenario Generation

The classified actions are translated into low-level sendevent commands. These commands are packaged into a script that can be executed on a target device using a modified version of the RERAN replay engine. This allows the recorded scenario to be replayed exactly as it was captured, without requiring the original app to be modified.

Evaluation and Results

We evaluated V2S+ using a comprehensive dataset of 243 videos containing over 4,000 GUI actions across 90+ popular native and hybrid apps.

  • Accuracy: The Faster R-CNN achieved approximately 97% precision in identifying touch locations.
  • Reproduction Success: V2S+ successfully reproduced 90.2% of sequential actions in native applications on physical devices. For hybrid applications on emulators, the success rate was 83%.
  • Bug Reproduction: In a targeted study of 20 apps, V2S+ successfully replayed 96.67% of recorded bugs and crashes, demonstrating its immediate utility for QA teams.
  • Efficiency: While processing a 3-minute video takes roughly 105 minutes, the process is entirely automated and can be offloaded to background servers, freeing developers from manual reproduction tasks.

Key Insights

One of the primary breakthroughs of V2S+ is its pixel-level universality. By ignoring the underlying code and focusing on visual indicators, it bypasses the limitations of metadata-based tools that struggle with hybrid apps. Additionally, the research highlighted a hardware-software dependency: for complex gestures like "flicks" to be accurately resolved, the video must be recorded at a minimum of 30 FPS.

Conclusion and Future Work

V2S+ demonstrates that computer vision can effectively automate the translation of visual artifacts into executable test scenarios. A case study involving industrial partners from Microsoft and 7-Eleven confirmed that the tool provides accurate scripts that are highly valued in professional debugging workflows.

Future iterations of V2S+ aim to address current limitations, such as resolution sensitivity (currently requiring models trained for specific aspect ratios) and coordinate dependence. By mapping touches to specific UI components rather than hardcoded coordinates, future versions could allow recordings from a phone to be replayed seamlessly on a tablet or different device models.

Summary updated

Abstract

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S+, an automated approach for translating video recordings of Android app usages into replayable scenarios. V2S+ is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user gestures captured in a video, and convert these into a replayable test scenario. Given that V2S+ takes a computer vision-based approach, it is applicable to both hybrid and native Android applications. We performed an extensive evaluation of V2S+ involving 243 videos depicting 4,028 GUI-based actions collected from users exercising features and reproducing bugs from a collection of over 90 popular native and hybrid Android apps. Our results illustrate that V2S+ can accurately replay scenarios from screen recordings, and is capable of reproducing $\approx$ 90.2% of sequential actions recorded in native application scenarios on physical devices, and $\approx$ 83% of sequential actions recorded in hybrid application scenarios on emulators, both with low overhead. A case study with three industrial partners illustrates the potential usefulness of V2S+ from the viewpoint of developers.
BibTeX
@article{bernalcardenas2023translating,
  author = {Carlos Bernal-Cárdenas and Nathan Cooper and Madeleine Havranek and Kevin Moran and Oscar Chaparro and Denys Poshyvanyk and Andrian Marcus},
  title = {{Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios}},
  journal = {IEEE Trans. Software Eng.},
  year = {2023},
  url = {https://openreview.net/forum?id=I5pLxRKhAG},
}
Figure