Enabling Mutation Testing for Android Apps

Mario Linares Vásquez, Gabriele Bavota, Michele Tufano, Kevin Moran, Massimiliano Di Penta, Christopher Vendome, Carlos Bernal-Cárdenas, Denys Poshyvanyk

Abstract

Mutation testing assesses test suite effectiveness by seeding intentional faults. However, traditional Java mutation operators often fail to capture Android-specific complexities, leading to high rates of invalid mutants and poor coverage of mobile-specific logic. This paper introduces MDroid+, a framework utilizing 38 specialized mutation operators derived from a comprehensive taxonomy of 262 Android fault types. Evaluation shows that MDroid+ significantly outperforms existing tools, achieving 62% coverage of real-world bug instances with a negligible stillborn rate of 0.56%.

Introduction

Mutation testing is a robust technique for evaluating test suite quality. By creating "mutants"—versions of the software containing small, intentional errors—developers can determine if their tests are capable of detecting these faults. While effective for general-purpose programming, traditional mutation tools like PIT or Major struggle with the Android ecosystem. Android apps rely on intricate lifecycles, resource-based configurations (XML), and hardware interactions that standard Java operators ignore. Consequently, these tools often produce "stillborn" mutants that fail to compile or "trivial" mutants that crash immediately, failing to represent the nuanced logic errors found in production mobile apps.

Methodology: A Taxonomy of Android Faults

To bridge the gap between general mutation testing and mobile-specific needs, the authors conducted a large-scale empirical study. They analyzed 2,023 software artifacts, including bug reports, commits, and Stack Overflow discussions, to identify recurring fault patterns unique to Android.

Figure 1: The comprehensive taxonomy of Android faults, categorized into 14 major groups such as Activities, GUI, and Non-functional Requirements.

This analysis resulted in a taxonomy of 262 fault types across 14 categories. From this foundation, 38 mutation operators were developed to simulate these real-world errors. Unlike traditional operators that modify arithmetic or logical symbols, MDroid+ operators target Android-specific components:

  • Resource Mutations: Modifying the AndroidManifest.xml to delete required activities.
  • API Mutations: Injecting invalid parameters into Android-specific APIs (e.g., null GPS coordinates).
  • Lifecycle Mutations: Introducing delays in the UI thread to trigger "Application Not Responding" (ANR) states.

Evaluation and Datasets

The framework was evaluated using a multi-faceted dataset:

  1. Taxonomy Source: Mined from 16,331 GitHub apps and over 700,000 Google Play reviews.
  2. Validation Set: 726 independent bug reports and commits were used to measure how well the new operators covered unseen real-world bugs.
  3. Target Apps: 55 open-source Android apps from the Androtest suite served as the testbed for the MDroid+ mutation engine.

Key Results

MDroid+ demonstrates a significant advancement in mutation testing for mobile applications. The study found that 35% of identified bugs are Android-specific, highlighting the necessity of domain-specific operators.

Figure 4: Coverage statistics for MDroid+ compared to other mutation tools, showing superior bug type and bug count coverage.

As shown in Figure 4, MDroid+ covers 38% of bug types and 62% of bug instances, nearly tripling the effectiveness of previous state-of-the-art tools like Deng et al. (15% type coverage). Furthermore, MDroid+ is highly efficient; its "stillborn" rate (mutants failing to compile) is a mere 0.56%, compared to 53.9% for muDroid. When combined with traditional Java tools, the hybrid approach reaches 73% coverage of real-world bug instances.

Conclusion

MDroid+ addresses the limitations of traditional mutation testing in the Android domain by grounding its operators in empirical fault data. By treating Android resources and lifecycles as first-class citizens, it provides a more accurate and efficient measure of test suite quality. Future work aims to automate the mutation of complex categories like multi-threading and to leverage these mutants for automated test case generation.

Summary updated

Abstract

Mutation testing has been widely used to assess the fault-detection effectiveness of a test suite, as well as to guide test case generation or prioritization. Empirical studies have shown that, while mutants are generally representative of real faults, an effective application of mutation testing requires "traditional" operators designed for programming languages to be augmented with operators specific to an application domain and/or technology. This paper proposes MDroid+, a framework for effective mutation testing of Android apps. First, we systematically devise a taxonomy of 262 types of Android faults grouped in 14 categories by manually analyzing 2,023 software artifacts from different sources (e.g., bug reports, commits). Then, we identified a set of 38 mutation operators, and implemented an infrastructure to automatically seed mutations in Android apps with 35 of the identified operators. The taxonomy and the proposed operators have been evaluated in terms of stillborn/trivial mutants generated and their capacity to represent real faults in Android apps, as compared to other well know mutation tools.
BibTeX
@inproceedings{vasquez2017enabling,
  author = {Mario Linares Vásquez and Gabriele Bavota and Michele Tufano and Kevin Moran and Massimiliano Di Penta and Christopher Vendome and Carlos Bernal-Cárdenas and Denys Poshyvanyk},
  title = {{Enabling Mutation Testing for Android Apps}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2017},
  url = {https://openreview.net/forum?id=rcIPhOxHZW5},
}
Figure