Abstract
Bug reports are essential for developers to confirm software problems, investigate their causes, and validate fixes. Unfortunately, reports often miss important information or are written unclearly, which can cause delays, increased issue resolution effort, or even the inability to solve issues. One of the most common components of reports that are problematic is the steps to reproduce the bug(s) (S2Rs), which are essential to replicate the described program failures and reason about fixes. Given the proclivity for deficiencies in reported S2Rs, prior work has proposed techniques that assist reporters in writing or assessing the quality of S2Rs. However, automated understanding of S2Rs is challenging, and requires linking nuanced natural language phrases with specific, semantically related program information. Prior techniques often struggle to form such language to program connections - due to issues in language variability and limitations of information gleaned from program analyses. To more effectively tackle the problem of S2R quality annotation, we propose a new technique called AstroBR, which leverages the language understanding capabilities of LLMs to identify and extract the S2Rs from bug reports and map them to GUI interactions in a program state model derived via dynamic analysis. We compared AstroBR to a related state-of-the-art approach and we found that AstroBR annotates S2Rs 25.2% better (in terms of F1 score) than the baseline. Additionally, AstroBR suggests more accurate missing S2Rs than the baseline (by 71.4% in terms of F1 score).
Introduction: The Challenge of Reproducing Bugs
When a user submits a bug report, the most critical piece of information is the Steps to Reproduce (S2R). If a developer cannot replicate the bug, they cannot fix it. Unfortunately, bug reports are frequently written in informal, ambiguous natural language. Users often omit "obvious" steps, use non-standard terminology (e.g., calling a "Submit" button a "Send" button), or describe actions that do not exist in the application's current UI.
Previous automated tools attempted to bridge this "language-to-program" gap using traditional Natural Language Processing (NLP), but they often struggled with the variability of human descriptions. Mapping a vague sentence like "I entered the info" to a specific sequence of clicks and text entries on a specific screen remains a significant technical hurdle.
The AstroBR Pipeline
To address these challenges, we introduce AstroBR, a four-phase pipeline designed to transform a text-heavy bug report into a verified execution path within the application.
The pipeline consists of the following stages:
- S2R Identification: Utilizing GPT-4, the tool scans a bug report to distinguish actual reproduction steps from other content, such as "Expected Behavior" or "Observed Behavior."
- Individual S2R Extraction: Complex sentences are broken down into atomic, machine-readable actions (e.g.,
[Click] [Login Button] [on] [Home Screen]). - App Execution Model Generation: Using CrashScope and manual traces, AstroBR constructs a directed graph of the application. Nodes represent GUI screens, and edges represent user interactions like taps, swipes, and text entries.
- Quality Assessment via Guided Traversal: AstroBR performs a Depth-First Search (DFS) on the app graph, using the LLM as a "navigator." At each screen, the LLM evaluates available UI components to find the best match for the user's written step.
AstroBR labels steps as Correct (CS), Ambiguous (AS) (matching multiple UI elements), or Vocabulary Mismatch (VM) (no match found). Crucially, if a gap exists between two reported steps, AstroBR identifies the shortest path in the graph to fill it, flagging these as Missing Steps (MS).
Evaluation and Results
AstroBR was evaluated using a test set of 21 bug reports from 6 Android applications, compared against the previous state-of-the-art tool, Euler. The ground truth was established through manual annotation of 1,031 sentences and 82 individual S2Rs.
The results demonstrate significant improvements:
- Quality Annotation: AstroBR achieved an F1 score of 0.879, representing a 25.2% relative improvement over Euler (0.702).
- Missing Step Identification: AstroBR identified missing steps with an F1 score of 0.639, outperforming Euler (0.373) by 71.4%.
- Precision: AstroBR suggested far fewer unnecessary steps (56 compared to Euler's 190), indicating higher reliability in its suggestions.
Key Insights
- Zero-Shot Performance: Interestingly, Zero-Shot (ZS) prompting outperformed Few-Shot or Chain-of-Thought methods. In complex bug reports, additional examples often introduced noise that led to false positives.
- Semantic Understanding: Unlike traditional keyword-matching tools, AstroBR leverages the semantic understanding of LLMs. It can recognize that "hit the plus sign" and "tap the add button" refer to the same UI interaction, even if the underlying code labels differ.
- Graph-Based Verification: The ability to suggest missing steps by traversing the application's execution graph is a major advantage over purely text-based analysis.
Limitations and Future Directions
AstroBR's effectiveness is currently tied to the coverage of its dynamic analysis; if a screen is not discovered during exploration, the tool cannot map steps to it. Additionally, it does not yet support complex gestures like multi-finger swipes or screen rotation. Future work may involve incorporating multimodal LLMs to process GUI images directly, providing richer context for mapping difficult UI elements.
Summary updated