This paper, presented at ESEC/FSE 2019, introduces Euler, an automated approach to assess the quality of "Steps to Reproduce" (S2Rs) in bug reports. By combining natural language processing with dynamic analysis of Android applications, Euler identifies missing, ambiguous, or incorrect steps, providing actionable feedback to bug reporters.
TL;DR
Euler is an automated tool that evaluates the quality of "Steps to Reproduce" in bug reports by matching natural language descriptions to an app's actual execution model. It identifies 98% of reported steps and correctly flags 73% of quality issues, helping bridge the communication gap between users and developers.
What Problem Does This Paper Address?
Bug reports are essential for software maintenance, but they are often plagued by low-quality "Steps to Reproduce" (S2Rs). When steps are missing, vague, or technically inaccurate, developers waste significant time trying to replicate the issue—or worse, they may be unable to fix the bug at all.
While researchers have tried to automate test case generation from bug reports, these tools often fail because the input text is too messy. The authors identify a "reproducibility crisis" and propose that the solution lies in providing immediate, automated feedback to the reporter at the moment they submit the bug.
Methods
Euler’s workflow consists of four main phases:
- S2R Identification: Euler uses a neural sequence labeling model (Bi-LSTM-CRF) to find sentences that describe reproduction steps. It then applies dependency parsing to extract structured tuples in the format:
[action] [object] [preposition] [object2]. - Execution Model Generation: The tool uses an automated "ripping" engine (based on CrashScope) to explore the Android app. It builds a directed graph where nodes are unique GUI screens and edges are user interactions (taps, swipes, etc.).
- S2R Resolution: This is the core "matching" logic. Euler attempts to map the textual steps to paths in the execution graph. It uses a custom similarity score based on Longest Common Substrings (LCS) to handle variations in how users describe GUI components.
- Quality Annotation: Based on the matching results, Euler generates a report with four types of annotations:
- High Quality (HQ): The step precisely matches an app interaction.
- Ambiguous Step (AS): The step matches multiple possible GUI components or events.
- Vocabulary Mismatch (VM): The step doesn't match any known application interaction.
- Missing Step (MS): Euler identifies "inferred steps" needed to reach the state described in the report.
Datasets
- Training Data: The word embeddings were trained on a massive corpus of 819,000 bug reports from 358 open-source projects.
- Evaluation Set: The authors sampled 24 bug reports from six popular open-source Android apps: Aard Dictionary, Droid Weight, GnuCash, Mileage, Schedule, and A Time Tracker.
- Human Baseline: To evaluate Euler, the authors recruited 12 external evaluators (including PhD students and professors) to assess the accuracy and usefulness of Euler's quality reports against "ideal" reproduction scenarios.
Key Insights and Ideas
- Bridging the Semantic Gap: The most striking idea is the mapping of human verbs to "Action Groups." By grouping "hit," "press," and "tap" into a single
CLICKcategory, Euler handles the inherent messiness of natural language. - Identifying the "Invisible": Euler doesn't just check what is written; it looks for what isn't. By finding the shortest path between two reported steps in the execution graph, it can explicitly tell a reporter: "You missed three steps between opening the app and clicking 'Save'."
- Context-Aware Matching: Euler prioritizes interactions that occur on screens "near" the current state in the execution graph, which helps disambiguate common terms like "Settings" or "Back."
Key Results
- High Recall: Euler correctly identified 98% of the existing steps to reproduce in the test reports.
- Accurate Feedback: 73% of the quality annotations (HQ, AS, VM, MS) produced by Euler were judged as correct by human evaluators.
- Missing Step Detection: The tool successfully identified 58% of the steps that reporters had accidentally omitted.
- Perceived Usefulness: 83% of evaluators agreed that Euler’s reports would help users write better bug reports, and 87% found the reports easy to understand.
Limitations and Future Work
- Exploration Limits: Euler is only as good as its execution model. If the automated "ripper" cannot reach a specific screen (e.g., due to a login wall or complex logic), Euler cannot validate steps on that screen.
- Complex Logic: The current version struggles with non-linear steps, such as "repeat this five times" or complex conditional logic ("if X happens, do Y").
- Vocabulary Sensitivity: While "Action Groups" help, "Vocabulary Mismatches" still occur if the user's terminology is vastly different from the internal labels used by developers.
- Future Directions: The authors plan to integrate more screenshots into the feedback reports and optimize the app exploration strategies to cover more "hidden" states of the software.
Summary updated