Assessing the Quality of the Steps to Reproduce in Bug Reports

Oscar Chaparro, Carlos Bernal-Cárdenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, Vincent Ng

Abstract

A major problem with user-written bug reports, indicated by developers and documented by researchers, is the (lack of high) quality of the reported steps to reproduce the bugs. Low-quality steps to reproduce lead to excessive manual effort spent on bug triage and resolution. This paper proposes Euler, an approach that automatically identifies and assesses the quality of the steps to reproduce in a bug report, providing feedback to the reporters, which they can use to improve the bug report. The feedback provided by Euler was assessed by external evaluators and the results indicate that Euler correctly identified 98% of the existing steps to reproduce and 58% of the missing ones, while 73% of its quality annotations are correct.

Introduction

Bug reports are the primary communication channel between users and developers for software maintenance. However, the utility of these reports is often compromised by low-quality "Steps to Reproduce" (S2Rs). When steps are missing, vague, or technically inaccurate, developers waste significant time trying to replicate the issue—or worse, they may be unable to fix the bug at all.

Presented at ESEC/FSE 2019, Euler is an automated approach designed to bridge this communication gap. By combining natural language processing (NLP) with dynamic analysis of Android applications, Euler identifies missing, ambiguous, or incorrect steps, providing actionable feedback to bug reporters at the moment of submission.

The Reproducibility Crisis in Bug Reporting

While researchers have previously attempted to automate test case generation directly from bug reports, these tools often fail because the input text is unstructured and "messy." The authors identify a "reproducibility crisis" where the burden of deciphering user intent falls heavily on developers. Euler proposes that the solution lies in providing immediate, automated feedback to the reporter, encouraging them to refine their report before it reaches the developer's queue.

Methodology

Euler’s workflow is divided into four distinct phases that transition from raw text to a structured quality assessment.

1. S2R Identification

Euler utilizes a neural sequence labeling model (Bi-LSTM-CRF) to isolate sentences within a bug report that describe reproduction steps. Following identification, dependency parsing is applied to extract structured tuples in the format: [action] [object] [preposition] [object2]. This step transforms natural language into a machine-readable format.

2. Execution Model Generation

To understand the application's capabilities, Euler employs an automated "ripping" engine based on CrashScope. This engine explores the Android application to build a directed graph—the execution model. In this graph, nodes represent unique GUI screens, and edges represent user interactions such as taps or swipes.

The execution model represents the app's states and transitions as a directed graph.

3. S2R Resolution

This phase involves matching the extracted textual steps to specific paths within the execution graph. Euler uses a custom similarity score based on Longest Common Substrings (LCS) to account for linguistic variations in how users describe GUI components. It prioritizes interactions occurring on screens "near" the current state in the execution graph, which helps disambiguate common terms like "Settings" or "Back."

4. Quality Annotation

Based on the matching results, Euler generates a comprehensive report. It identifies four primary types of quality issues:

  • High Quality (HQ): The step precisely matches an app interaction.
  • Ambiguous Step (AS): The step matches multiple possible GUI components or events.
  • Vocabulary Mismatch (VM): The step doesn't match any known application interaction.
  • Missing Step (MS): Euler identifies "inferred steps" required to reach the state described in the report.
Euler generates a Quality Report that annotates each step with specific quality indicators.

Evaluation and Results

The authors evaluated Euler using a massive corpus of 819,000 bug reports for training word embeddings. The core evaluation was conducted on 24 bug reports from six popular open-source Android apps. Twelve external evaluators, including PhD students and professors, assessed Euler's output against "ideal" reproduction scenarios.

The results demonstrate high efficacy:

  • Recall: Euler correctly identified 98% of existing S2Rs.
  • Annotation Accuracy: 73% of the quality annotations (HQ, AS, VM, MS) were judged correct by human experts.
  • Inference: The tool successfully identified 58% of steps that reporters had omitted.
  • User Perception: 83% of evaluators agreed that Euler’s reports would help users write better bug reports, and 87% found the feedback easy to understand.

Discussion

A key innovation in Euler is its ability to bridge the semantic gap by mapping human verbs to "Action Groups" (e.g., grouping "hit," "press," and "tap" into a CLICK category). Furthermore, by finding the shortest path between two reported steps in the execution graph, Euler can identify "invisible" missing steps, explicitly informing the reporter of gaps in their logic.

However, Euler is not without limitations. Its accuracy is tethered to the quality of the execution model; if the automated ripper cannot bypass a login wall or reach a complex state, Euler cannot validate steps on those screens. Additionally, the tool currently struggles with non-linear logic, such as loops or conditional steps.

Conclusion

Euler represents a significant step toward automating bug report triage. By providing real-time feedback to reporters, it reduces the manual effort required by developers to reproduce bugs. Future work aims to integrate visual aids like screenshots into the feedback reports and optimize exploration strategies to cover more "hidden" software states.

Summary updated

Abstract

A major problem with user-written bug reports, indicated by developers and documented by researchers, is the (lack of high) quality of the reported steps to reproduce the bugs. Low-quality steps to reproduce lead to excessive manual effort spent on bug triage and resolution. This paper proposes Euler, an approach that automatically identifies and assesses the quality of the steps to reproduce in a bug report, providing feedback to the reporters, which they can use to improve the bug report. The feedback provided by Euler was assessed by external evaluators and the results indicate that Euler correctly identified 98% of the existing steps to reproduce and 58% of the missing ones, while 73% of its quality annotations are correct.
BibTeX
@inproceedings{chaparro2019assessing,
  author = {Oscar Chaparro and Carlos Bernal-Cárdenas and Jing Lu and Kevin Moran and Andrian Marcus and Massimiliano Di Penta and Denys Poshyvanyk and Vincent Ng},
  title = {{Assessing the Quality of the Steps to Reproduce in Bug Reports}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2019},
  url = {https://openreview.net/forum?id=SVn-hdlr-b5},
}
Figure