TL;DR
AstroBR is a new tool that combines the linguistic power of GPT-4 with dynamic app analysis to automatically evaluate "Steps to Reproduce" (S2Rs) in bug reports. It outperforms previous state-of-the-art methods by over 25% in quality annotation and is significantly better at identifying and suggesting missing steps that users often forget to include.
What Problem Does This Paper Address?
When a user submits a bug report, the most critical piece of information is the Steps to Reproduce (S2R). If a developer can’t replicate the bug, they can’t fix it. Unfortunately, most bug reports are written in messy, informal natural language. Users often skip "obvious" steps, use ambiguous terminology (calling a "Submit" button a "Send" button), or describe actions that don't actually exist in the app's current UI.
Previous automated tools tried to bridge this "language $\leftrightarrow$ program" gap using traditional Natural Language Processing (NLP), but they often stumbled over the sheer variety of ways humans describe software interactions. The challenge is mapping a vague sentence like "I entered the info" to a specific sequence of clicks and text entries on a specific screen.
Methods
The authors introduce AstroBR, a four-phase pipeline designed to turn a text-heavy bug report into a verified execution path:
- S2R Identification: Using GPT-4, the tool scans a bug report to distinguish actual reproduction steps from other content like "Expected Behavior" or "Observed Behavior."
- Individual S2R Extraction: It breaks down complex sentences into atomic, machine-readable actions (e.g.,
[Click] [Login Button] [on] [Home Screen]). - App Execution Model Generation: Using a tool called CrashScope and manual traces, AstroBR builds a directed graph of the app. Nodes represent GUI screens, and edges represent user interactions (taps, swipes, typing).
- Quality Assessment via Guided Traversal: This is the "secret sauce." AstroBR performs a Depth-First Search (DFS) on the app graph, but it uses GPT-4 as a "navigator." At each screen, the LLM looks at the available UI components and decides which one best matches the user's written step.
Through this process, AstroBR labels steps as Correct (CS), Ambiguous (AS) (matches multiple UI elements), or Vocabulary Mismatch (VM) (no match found). Crucially, if there is a gap between two reported steps, AstroBR finds the shortest path in the graph to fill it, identifying Missing Steps (MS).
Datasets
The researchers used two primary datasets:
- Development Set: 54 bug reports from 31 Android apps (e.g., finance trackers, web browsers) used to refine the LLM prompts.
- Test Set: 21 bug reports from 6 Android apps, originally used to evaluate the previous state-of-the-art tool, Euler.
- Ground Truth: The authors manually annotated 1,031 sentences and mapped 82 individual S2Rs to GUI interactions to create a gold standard for evaluation.
Key Insights and Ideas
- Zero-Shot Wins the Race: Surprisingly, the researchers found that Zero-Shot (ZS) prompting performed better than Few-Shot or Chain-of-Thought prompting. In the context of long, complex bug reports, adding extra examples to the prompt actually introduced "noise" that led to more false positives.
- The "Missing Step" Breakthrough: Identifying what isn't there is much harder than identifying what is. AstroBR’s ability to suggest missing steps (like "Open the side menu first") by traversing the app's execution graph is its most significant advantage over purely text-based tools.
- LLMs as Semantic Bridges: Traditional tools rely on keyword matching. AstroBR succeeds because GPT-4 understands that "hit the plus sign" and "tap the add button" refer to the same UI interaction, even if the underlying code uses different labels.
Key Results
AstroBR was compared directly against Euler, the previous leading approach:
- Overall Quality Annotation: AstroBR achieved an F1 score of 0.879, a 25.2% relative improvement over Euler (0.702).
- Missing Step Identification: This was the biggest win. AstroBR identified missing steps with an F1 score of 0.639, outperforming Euler (0.373) by 71.4%.
- Precision in Suggestions: AstroBR produced far fewer "hallucinated" or unnecessary steps. While Euler suggested 190 unnecessary steps, AstroBR only suggested 56.
Limitations and Future Work
While powerful, AstroBR has a few hurdles to clear:
- Dynamic Analysis Coverage: The tool is only as good as its map. If the automated exploration (CrashScope) fails to find a specific screen, AstroBR cannot map a user's step to it.
- Interaction Variety: The current version does not support complex gestures like screen rotation or multi-finger gestures.
- Cost and Latency: Relying on GPT-4 for every step of a graph traversal can be slow and expensive compared to local, specialized models.
The authors suggest that future versions could incorporate GUI images directly into the prompts (using multimodal LLMs) to provide even better context for mapping tricky UI elements.
Summary updated