Testing Criteria for Mobile Apps Based on Callback Sequences

Danilo Dominguez Perez, Wei LearXiv 2019 · OpenReview

TL;DR

This paper introduces three new white-box testing criteria for Android apps that focus on callback sequences, addressing the limitations of existing GUI-centric and structural coverage methods. By analyzing callback interactions, their approach helps uncover bugs that are often missed by traditional testing, leading to faster bug detection and improved app quality.

What Problem Does This Paper Address?

Mobile app quality is crucial for user adoption, and testing is a primary method for ensuring this quality. However, current testing approaches often rely heavily on structural coverage (like statement and branch coverage) or GUI event sequences. This overlooks a significant portion of mobile app behavior, particularly the complex interactions involving background tasks (Services, AsyncTasks), sensor events (GPS, camera), and the asynchronous nature of many API calls. These interactions, especially data sharing between callbacks, are a common source of bugs. The paper aims to bridge this gap by proposing testing criteria that specifically target these under-tested callback sequences.

Methods

The core of this paper lies in defining and evaluating three novel white-box testing criteria based on callback sequences:

  1. Event-Event Criterion: This criterion focuses on covering interactions between event handlers, including those for GUI events and other external events like sensor data. It ensures that sequences of two consecutive event callbacks are tested.
  2. Event-API Sync Criterion: This criterion targets synchronous callbacks invoked within API methods. It aims to cover sequences where an event handler is followed by a synchronous API callback, or where two synchronous callbacks within the same API method are executed sequentially.
  3. Event-API Async Criterion: This criterion addresses concurrency bugs by focusing on interleavings between event handlers and asynchronous callbacks invoked by API methods. It specifically looks for sequences involving two event callbacks and one asynchronous API callback.

To implement and measure these criteria, the authors utilize a Callback Control Flow Automata (CCFA) model. This model, derived from static analysis of the app's bytecode, represents all possible callback sequences. They then develop algorithms to generate the "ground truth" sequences that should be covered for each criterion. For testing, they instrument the apps to record actual callback execution traces. Finally, they compare these traces against the ground truth sequences to calculate coverage.

Datasets

The paper evaluates its approach on 15 open-source Android applications sourced from the F-Droid repository. These apps cover a range of categories and sizes. The authors also reference a pilot study that analyzed 1526 bugs from 6 apps to understand the types of callback interactions that lead to bugs.

Key Insights and Ideas

  • Callback interactions are a critical bug source: The research strongly suggests that many bugs arise from how different callbacks interact, especially when data is shared between them. This is a significant departure from solely focusing on individual callback logic or GUI flow.
  • Existing criteria miss crucial behaviors: Standard GUI testing and structural coverage are insufficient because they don't adequately capture the complex interplay of background tasks, sensor events, and asynchronous API calls.
  • Prioritizing neighbor callback interactions: The studies indicate that focusing on short sequences (length two) of interacting callbacks is highly effective for finding bugs, as data sharing and race conditions often occur between adjacent callbacks.
  • Concurrency bugs are prevalent and hard to find: The event-API async criterion is particularly important for uncovering race conditions, which are a common and difficult class of bugs in concurrent mobile applications.
  • CCFA as a powerful modeling tool: The Callback Control Flow Automata provides a robust way to represent and analyze the complex callback execution paths in Android apps, enabling the definition and measurement of new coverage criteria.

Key Results

  • Bug Detection: Testing guided by the proposed criteria found 17 bugs, which is 3 more bugs than the state-of-the-art tool Monkey.
  • Bug Detection Speed: The authors' testing approach was 7 times faster on average than Monkey in triggering the same set of bugs.
  • Coverage Improvement: For 12 out of 15 apps, the proposed criteria achieved better or equal coverage compared to Monkey, especially for event-driven behaviors that Monkey struggles to trigger.
  • Correlation with Bugs: A significant portion of bugs (30 out of 31) were correlated with increases in coverage for at least one of the proposed criteria. The event-event criterion showed the strongest correlation with bug discovery.
  • Concurrency Bug Focus: The event-API async criterion was found to be crucial for uncovering race condition bugs, with 8 such bugs identified and correlated with this criterion's coverage.

Limitations and Future Work

  • Manual Test Case Generation: The current approach relies on manually constructed test cases guided by the criteria. The authors acknowledge that developing automatic test generation tools based on these criteria is a key area for future work.
  • Feasible Sequences: Not all callback sequences generated by the CCFA are feasible in practice. The authors note that achieving 100% coverage for their criteria can be challenging due to these feasibility constraints.
  • Specialized Inputs: Some bugs missed by their testing were related to network connectivity events or Out-of-Memory exceptions, which require long-running tests or specific environmental conditions that were not fully automated in this study.
  • Tooling and App Compatibility: The implementation of CCFA generation and instrumentation relies on specific tools (Soot, Gator, PCS), which can limit the set of apps that can be analyzed.

Future work will focus on developing automated input generation tools, potentially integrating with existing frameworks like Stoat, to improve the efficiency and scalability of testing guided by these new criteria.

Summary updated

Abstract

App quality has been shown to be the most important indicator of app adoption. To assure quality, developers mainly use testing to find bugs in app and apply structural and GUI test coverage criteria. However, mobile apps have more behaviors than the GUI actions, e.g. an app also handles events from sensors and executes long-running background tasks through Android API calls to Services and AsyncTasks. Our studies found that there are important app behaviors via callback interactions that should be covered in testing, as data sharing between callbacks is common and is the cause of many existing bugs. We design a family of test criteria based on callback sequences and use the Callback Control Flow Automata (CCFA) to measure the coverage for testing. Our experiments show that guiding by our criteria, testing can find more bugs and trigger bugs faster than the state-of-the-art tools.
BibTeX
@misc{perez2019testing,
  author = {Danilo Dominguez Perez and Wei Le},
  title = {{Testing Criteria for Mobile Apps Based on Callback Sequences}},
  year = {2019},
  eprint = {1911.09201},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.09201},
}
Figure