Combining Language and App Ui Analysis for the Automated Assessment of Bug Reproduction Steps

Junayed Mahmud, Antu Saha, Oscar Chaparro, Kevin Moran, Andrian Marcus

TL;DR

AstroBR is a new tool that combines the linguistic power of GPT-4 with dynamic app analysis to automatically evaluate "Steps to Reproduce" (S2Rs) in bug reports. It outperforms previous state-of-the-art methods by over 25% in quality annotation and is significantly better at identifying and suggesting missing steps that users often forget to include.

What Problem Does This Paper Address?

When a user submits a bug report, the most critical piece of information is the Steps to Reproduce (S2R). If a developer can’t replicate the bug, they can’t fix it. Unfortunately, most bug reports are written in messy, informal natural language. Users often skip "obvious" steps, use ambiguous terminology (calling a "Submit" button a "Send" button), or describe actions that don't actually exist in the app's current UI.

Previous automated tools tried to bridge this "language $\leftrightarrow$ program" gap using traditional Natural Language Processing (NLP), but they often stumbled over the sheer variety of ways humans describe software interactions. The challenge is mapping a vague sentence like "I entered the info" to a specific sequence of clicks and text entries on a specific screen.

Methods

The authors introduce AstroBR, a four-phase pipeline designed to turn a text-heavy bug report into a verified execution path:

  1. S2R Identification: Using GPT-4, the tool scans a bug report to distinguish actual reproduction steps from other content like "Expected Behavior" or "Observed Behavior."
  2. Individual S2R Extraction: It breaks down complex sentences into atomic, machine-readable actions (e.g., [Click] [Login Button] [on] [Home Screen]).
  3. App Execution Model Generation: Using a tool called CrashScope and manual traces, AstroBR builds a directed graph of the app. Nodes represent GUI screens, and edges represent user interactions (taps, swipes, typing).
  4. Quality Assessment via Guided Traversal: This is the "secret sauce." AstroBR performs a Depth-First Search (DFS) on the app graph, but it uses GPT-4 as a "navigator." At each screen, the LLM looks at the available UI components and decides which one best matches the user's written step.

Through this process, AstroBR labels steps as Correct (CS), Ambiguous (AS) (matches multiple UI elements), or Vocabulary Mismatch (VM) (no match found). Crucially, if there is a gap between two reported steps, AstroBR finds the shortest path in the graph to fill it, identifying Missing Steps (MS).

Datasets

The researchers used two primary datasets:

  • Development Set: 54 bug reports from 31 Android apps (e.g., finance trackers, web browsers) used to refine the LLM prompts.
  • Test Set: 21 bug reports from 6 Android apps, originally used to evaluate the previous state-of-the-art tool, Euler.
  • Ground Truth: The authors manually annotated 1,031 sentences and mapped 82 individual S2Rs to GUI interactions to create a gold standard for evaluation.

Key Insights and Ideas

  • Zero-Shot Wins the Race: Surprisingly, the researchers found that Zero-Shot (ZS) prompting performed better than Few-Shot or Chain-of-Thought prompting. In the context of long, complex bug reports, adding extra examples to the prompt actually introduced "noise" that led to more false positives.
  • The "Missing Step" Breakthrough: Identifying what isn't there is much harder than identifying what is. AstroBR’s ability to suggest missing steps (like "Open the side menu first") by traversing the app's execution graph is its most significant advantage over purely text-based tools.
  • LLMs as Semantic Bridges: Traditional tools rely on keyword matching. AstroBR succeeds because GPT-4 understands that "hit the plus sign" and "tap the add button" refer to the same UI interaction, even if the underlying code uses different labels.

Key Results

AstroBR was compared directly against Euler, the previous leading approach:

  • Overall Quality Annotation: AstroBR achieved an F1 score of 0.879, a 25.2% relative improvement over Euler (0.702).
  • Missing Step Identification: This was the biggest win. AstroBR identified missing steps with an F1 score of 0.639, outperforming Euler (0.373) by 71.4%.
  • Precision in Suggestions: AstroBR produced far fewer "hallucinated" or unnecessary steps. While Euler suggested 190 unnecessary steps, AstroBR only suggested 56.

Limitations and Future Work

While powerful, AstroBR has a few hurdles to clear:

  • Dynamic Analysis Coverage: The tool is only as good as its map. If the automated exploration (CrashScope) fails to find a specific screen, AstroBR cannot map a user's step to it.
  • Interaction Variety: The current version does not support complex gestures like screen rotation or multi-finger gestures.
  • Cost and Latency: Relying on GPT-4 for every step of a graph traversal can be slow and expensive compared to local, specialized models.

The authors suggest that future versions could incorporate GUI images directly into the prompts (using multimodal LLMs) to provide even better context for mapping tricky UI elements.

Summary updated

Abstract

Bug reports are essential for developers to confirm software problems, investigate their causes, and validate fixes. Unfortunately, reports often miss important information or are written unclearly, which can cause delays, increased issue resolution effort, or even the inability to solve issues. One of the most common components of reports that are problematic is the steps to reproduce the bug(s) (S2Rs), which are essential to replicate the described program failures and reason about fixes. Given the proclivity for deficiencies in reported S2Rs, prior work has proposed techniques that assist reporters in writing or assessing the quality of S2Rs. However, automated understanding of S2Rs is challenging, and requires linking nuanced natural language phrases with specific, semantically related program information. Prior techniques often struggle to form such language to program connections - due to issues in language variability and limitations of information gleaned from program analyses. To more effectively tackle the problem of S2R quality annotation, we propose a new technique called AstroBR, which leverages the language understanding capabilities of LLMs to identify and extract the S2Rs from bug reports and map them to GUI interactions in a program state model derived via dynamic analysis. We compared AstroBR to a related state-of-the-art approach and we found that AstroBR annotates S2Rs 25.2% better (in terms of F1 score) than the baseline. Additionally, AstroBR suggests more accurate missing S2Rs than the baseline (by 71.4% in terms of F1 score).
BibTeX
@inproceedings{mahmud2025combining,
  author = {Junayed Mahmud and Antu Saha and Oscar Chaparro and Kevin Moran and Andrian Marcus},
  title = {{Combining Language and App Ui Analysis for the Automated Assessment of Bug Reproduction Steps}},
  booktitle = {ICPC},
  year = {2025},
  doi = {10.1109/icpc66645.2025.00010},
}

Loading related content...

Figure