Abstract
Many software bugs are reported manually, particularly bugs that manifest themselves visually in the user interface. End-users typically report these bugs via app reviewing websites, issue trackers, or in-app built-in bug reporting tools, if available. While these systems have various features that facilitate bug reporting (e.g., textual templates or forms), they often provide limited guidance, concrete feedback, or quality verification to end-users, who are often inexperienced at reporting bugs and submit low-quality bug reports that lead to excessive developer effort in bug report management tasks. We propose an interactive bug reporting system for end-users (Burt), implemented as a task-oriented chatbot. Unlike existing bug reporting systems, Burt provides guided reporting of essential bug report elements (i.e., the observed behavior, expected behavior, and steps to reproduce the bug), instant quality verification, and graphical suggestions for these elements. We implemented a version of Burt for Android and conducted an empirical evaluation study with end-users, who reported 12 bugs from six Android apps studied in prior work. The reporters found that Burt's guidance and automated suggestions/clarifications are useful and Burt is easy to use. We found that Burt reports contain higher-quality information than reports collected via a template-based bug reporting system. Improvements to Burt, informed by the reporters, include support for various wordings to describe bug report elements and improved quality verification. Our work marks an important paradigm shift from static to interactive bug reporting for end-users.
Introduction
There is a persistent "knowledge gap" between software end-users and developers. To effectively resolve a bug, developers require three critical pieces of information: the Observed Behavior (OB), the Expected Behavior (EB), and the Steps to Reproduce (S2R). However, end-users—who are often the first to encounter visual or functional issues—frequently submit vague or incomplete reports (e.g., "the app crashed") via static forms or app store reviews.
Traditional reporting tools are "static"; they do not inform the user if their description is ambiguous or if a crucial step has been omitted until a developer reviews the report days or weeks later. This delay leads to an inefficient "ping-pong" of communication. This paper introduces BURT (BUg RepoRTing), an interactive chatbot designed to bridge this gap by transforming bug reporting into a real-time, guided conversation.
The BURT System
BURT replaces static bug-reporting forms with a task-oriented chatbot that guides users through the reporting process. The system is built upon several sophisticated back-end components:
- App Execution Model: Before deployment, BURT builds a "map" of the target application. This directed weighted graph represents app screens as nodes and user actions (taps, swipes, etc.) as edges. This model is constructed using a combination of automated "ripping" (via CrashScope) and crowdsourced usage traces.
- Natural Language Parser (NL): When a user describes an action, BURT uses dependency parsing to extract "subject-action-object" tuples. For instance, "I tapped the save button" is converted into a structured format the system can validate against the app model.
- Dialogue Manager (DM): This component manages the conversation flow, ensuring the user provides the OB, EB, and S2Rs in a logical sequence.
- Real-time Quality Verification: As the user describes each step, BURT attempts to match the input against the App Execution Model. If the description is ambiguous or describes an impossible state, BURT requests immediate clarification.
- Predictive Suggestions: If BURT can identify the user's current state within the app, it provides visual suggestions for the next likely steps using screenshots. Users can confirm a step by clicking an image rather than typing a description.
Empirical Evaluation
The researchers evaluated BURT using 12 real-world bugs from six diverse Android applications, including AntennaPod, GnuCash, and Time Tracker. The study involved 36 participants divided into two groups: one using BURT and a control group using a traditional template-based web form (ITRAC).
Key Results
- Improved Report Quality: Reports generated via BURT contained significantly higher-quality information. Specifically, BURT reports had only 8.3% incorrect steps compared to 20.4% in the control group. Missing steps were also reduced from 32.0% to 19.4%.
- Matching Accuracy: BURT successfully matched user descriptions to app states in 60-76% of cases, demonstrating the robustness of the NL parser and execution model.
- User Experience: Participants generally found the chatbot intuitive. 12 out of 18 users in the BURT group rated the system as "easy" or "somewhat easy" to use, praising the step-by-step guidance.
Discussion and Insights
The study revealed that BURT’s proactive guidance serves different user demographics in unique ways. Novice reporters benefited most from accuracy guidance, which prevented them from describing "hallucinated" or incorrect steps. In contrast, experienced reporters benefited from completeness guidance, helping them include "obvious" steps they might otherwise omit.
Furthermore, the use of visual confirmation proved highly effective. Users found it cognitively easier to recognize a screenshot of a screen they had recently interacted with than to describe that screen in technical terms. By forcing users to reconcile their natural language with the app's actual GUI components, BURT effectively translates "user-speak" into actionable "developer-speak."
Limitations and Future Work
Despite its success, BURT faces a "cold start" challenge: it requires a pre-built execution model. If a bug occurs in an unmapped section of an app, the system's effectiveness is diminished. Additionally, while the NL parser is robust, it can struggle with non-standard phrasing, occasionally forcing users to paraphrase their descriptions to satisfy the bot.
Future research will focus on expanding BURT’s vocabulary support and investigating how interactive reports impact the actual time-to-fix for developers. Transitioning from static to interactive reporting represents a significant paradigm shift that could drastically reduce the manual effort involved in bug triage and resolution.
Summary updated