Abstract
Software maintenance constitutes a large portion of the software development lifecycle. To carry out maintenance tasks, developers often need to understand and reproduce bug reports. As such, there has been increasing research activity coalescing around the notion of automating various activities related to bug reporting. A sizable portion of this research interest has focused on the domain of mobile apps. However, as research around mobile app bug reporting progresses, there is a clear need for a manually vetted and reproducible set of real-world bug reports that can serve as a benchmark for future work. This paper presents ANDROR2: a dataset of 90 manually reproduced bug reports for Android apps listed on Google Play and hosted on GitHub, systematically collected via an in-depth analysis of 459 reports extracted from the GitHub issue tracker. For each reproduced report, ANDROR2 includes the original bug report, an apk file for the buggy version of the app, an executable reproduction script, and metadata regarding the quality of the reproduction steps associated with the original report. We believe that the ANDROR2 dataset can be used to facilitate research in automatically analyzing, understanding, reproducing, localizing, and fixing bugs for mobile applications as well as other software maintenance activities more broadly.
Introduction
In software development, reproducing bug reports is often the most time-consuming part of maintenance, yet it remains an essential first step for any fix. The "reproducibility crisis" in software engineering presents a major hurdle: while researchers develop AI and program analysis tools to automatically reproduce, locate, and fix bugs, they lack a reliable "gold standard" to test them against.
Existing mobile bug datasets are often limited in scope, focusing almost exclusively on app crashes—which are easier to detect but represent only a fraction of real-world issues. Furthermore, many lack the necessary artifacts, such as specific APK versions, required to execute the code. ANDROR2 addresses this gap by providing a systematically curated set of real-world bugs, including complex, non-crashing functional errors.
Methodology
The authors followed a rigorous three-phase methodology to ensure the dataset's quality and utility for the research community:
- Collection: The team mined over 82,000 issues from GitHub repositories of Java-based Android apps. To ensure relevance, these apps were cross-referenced with the Google Play Store.
- Filtering: The list was narrowed to 459 promising candidates by selecting reports from non-contributors (simulating end-user experiences) that explicitly mentioned "steps" to reproduce the bug.
- Reproduction: This phase involved significant manual effort, with five authors spending roughly six man-months reproducing the bugs. Each bug was verified by at least two researchers. For every successful reproduction, an automated script was created using the UIAutomator framework.
The ANDROR2 Dataset
The primary contribution of this work is the ANDROR2 dataset, which offers a robust benchmark for automated bug reproduction and localization tools. The dataset includes:
- 90 Reproducible Bug Reports: Comprising 23 crashes and 67 non-crashing failures (e.g., UI glitches or incorrect logic).
- Target APKs: The specific buggy versions of the applications required for reproduction.
- Reproduction Scripts: Executable UIAutomator tests that trigger the reported bug.
- Comprehensive Metadata: JSON files detailing the number of steps, required environment setups, and discrepancies found in the original user reports.
ANDROR2 is significantly larger and more diverse than previous benchmarks like FUSION (15 reports) or YAKUSU (48 reports), which focused primarily on crashes.
Key Insights and Results
The metadata provided in ANDROR2 offers granular insights into the nature of mobile app bugs and the challenges of manual reproduction:
- The "S2R" Gap: User-provided "Steps to Reproduce" (S2Rs) are often unreliable. The authors found that 33 of the 90 bugs required specific GUI actions that the original reporters failed to mention.
- Setup Requirements: 77 out of the 90 bugs required "setup" actions—such as logging in, creating a specific file, or changing a setting—before reproduction steps could begin. This explains why simple record-and-replay tools often fail.
- Failure Diversity: The dataset includes 23 crashes, 34 output errors, and 33 GUI errors. By including 67 non-crashing failures, the authors provide a testbed for "oracle" research—tools that determine if a program is behaving incorrectly even without a crash.
- Complexity: On average, reproducing a bug required 14 GUI actions: 8 setup actions and 6 actions specific to the bug's sequence.
Conclusion
While the manual effort required for reproduction is high—only 90 of 459 analyzed issues were successfully packaged—ANDROR2 represents a significant step toward more realistic evaluation of automated maintenance tools. Many failures could not be reproduced due to dependencies on specific hardware, private accounts, or inaccessible external files.
Future work may involve adding formal assertions to the reproduction scripts to create a full functional test suite or creating script variations to help evaluate test case prioritization and duplicate detection tools. ANDROR2 is available for the research community on Zenodo and GitHub.
Summary updated