This paper, presented at ESEC/FSE ’22, introduces BURT (BUg RepoRTing), an interactive chatbot designed to help end-users submit high-quality bug reports for Android applications.
TL;DR
BURT replaces static bug-reporting forms with a task-oriented chatbot that guides users through the reporting process in real-time. By matching user descriptions to a pre-built model of the app's behavior, BURT provides instant quality feedback and visual suggestions, significantly reducing missing or incorrect steps in bug reports.
What Problem Does This Paper Address?
There is a persistent "knowledge gap" between software end-users and developers. Developers need specific information to fix a bug: the Observed Behavior (OB), the Expected Behavior (EB), and the Steps to Reproduce (S2R). End-users, however, often submit vague reports (e.g., "the app crashed") via static forms or app store reviews.
Current reporting tools are "static"—they don't tell the user if their description is ambiguous or if they've skipped a crucial step until a developer looks at it days later. This leads to a "ping-pong" of comments between users and developers, wasting time and resources. BURT aims to bridge this gap by making the reporting process interactive and self-correcting.
Methods
The authors developed BURT as a web-based chatbot with several sophisticated back-end components:
- App Execution Model: Before BURT is deployed for a specific app, the system builds a "map" of the app. This is a directed weighted graph where nodes are app screens and edges are user actions (taps, swipes, etc.). This model is built using a combination of automated "ripping" (using a tool called CrashScope) and crowdsourced usage traces.
- Natural Language Parser (NL): When a user types a description, BURT uses dependency parsing (via Stanford CoreNLP) to extract "subject-action-object" tuples. For example, "I tapped the save button" is parsed into a structured format the system can understand.
- Dialogue Manager (DM): This component controls the conversation flow, ensuring the user provides the OB, EB, and S2Rs in order.
- Real-time Quality Verification: As the user describes a step, BURT attempts to match it against the App Execution Model. If the description is ambiguous or doesn't match any known app state, BURT asks for clarification immediately.
- Predictive Suggestions: If BURT knows where the user is in the app, it can suggest the next likely steps using screenshots. The user can simply click a screenshot to confirm a step rather than typing it out.
Datasets
The researchers evaluated BURT using:
- 12 Real-world Android Bugs: Selected from a known dataset (Cooper et al.) across six diverse apps: AntennaPod (podcasts), Time Tracker, Android Token (2FA), GnuCash (finance), GrowTracker (plants), and Droid Weight.
- User Study Groups: 36 participants in total. 18 used BURT, and 18 used a traditional template-based web form (ITRAC) for comparison.
- App Execution Data: Generated via automated exploration and manual traces to populate the underlying graph models for the six apps.
Key Insights and Ideas
- The Power of Proactive Guidance: Unlike static forms, BURT’s ability to say "I don't understand that step" while the user is still reporting prevents low-quality data from ever reaching the developer.
- Visual Confirmation over Text: Users found the screenshot suggestions highly effective. It’s cognitively easier to recognize a screen you just saw than to describe it in technical terms.
- Experience-Dependent Benefits: The study found a fascinating split in how BURT helps:
- Novices benefited most from accuracy guidance, helping them avoid describing "incorrect" steps that don't actually exist in the app.
- Experienced reporters benefited most from completeness guidance, helping them avoid "missing" steps that they might otherwise take for granted.
- Bridging the Vocabulary Gap: By forcing the user to reconcile their language with the app's actual GUI components (via the execution model), BURT translates "user-speak" into "developer-speak."
Key Results
- Higher Quality Reports: Reports generated by BURT contained significantly fewer incorrect steps (8.3% vs. 20.4% in the control group) and fewer missing steps (19.4% vs. 32.0%).
- High Accuracy: BURT was able to correctly match user descriptions to app states in approximately 60-76% of cases, depending on the element being reported.
- User Preference: Users generally found BURT easy to use and praised the "step-by-step" guidance. 12 out of 18 users rated it as "easy" or "somewhat easy" to use.
Limitations and Future Work
- The "Cold Start" Problem: BURT requires a pre-built execution model. If a bug occurs in a part of the app that wasn't "mapped" during the initial exploration, BURT's effectiveness drops.
- Vocabulary Constraints: While the NL parser is robust, it still struggles with highly creative or non-standard phrasing. Users sometimes felt they had to "paraphrase" to satisfy the bot.
- Bug Complexity: The study focused on visual/functional bugs. It remains to be seen how BURT would handle "silent" bugs (like data sync issues) that don't have a clear visual manifestation.
- Future Directions: The authors suggest expanding BURT to support more diverse wordings and investigating how these interactive reports actually impact the time it takes for a developer to fix the bug.
Summary updated