Abstract
Current AI coding assistants often provide superficial fixes due to a lack of context and a "one-shot" interaction model. This paper introduces Robin, a debugging assistant that utilizes "insert expansion" and guided turn-taking to engage in multi-turn collaborative troubleshooting. Evaluated through a user study with 12 industry professionals, Robin demonstrated a 5x improvement in bug resolution rates and a 2x improvement in bug localization compared to standard chat-based baselines by shifting the interaction from immediate answers to investigative dialogue.
Introduction
While Large Language Models (LLMs) are now integrated into most IDEs, their interaction model remains largely stuck in a "unilateral question-answer" loop. The developer asks a question, and the AI attempts to provide a final answer immediately. This approach leads to two major issues: pre-mature bug localization, where AIs provide "band-aid" fixes for symptoms rather than root causes, and high cognitive load, placing the burden of investigation entirely on the developer. The researchers aim to align AI assistants with Gricean maxims of conversation—specifically quality (avoiding speculation) and quantity (providing the right amount of info at the right time).
Methodology
Robin is an extension for Visual Studio that implements a specialized agent-based workflow to move beyond one-shot responses. The system architecture consists of several key components designed to foster collaboration:
- Hardness Classifier: Analyzes the bug and context to decide if it's a "one-shot" fix (Eager) or requires a "collaborative" approach.
- Insert Expansion Pattern: Instead of answering immediately, Robin initiates auxiliary turns to ask the user for missing information, such as variable values or specific method code.
- Collaborative Responder: Guides the user through debugging workflows, suggesting specific places to set breakpoints or variables to watch.
- Follow-up Generator: Creates context-aware prompts that act as a style guide for the user, making it easier to continue the investigative dialogue.
Evaluation
The system was evaluated using real-world bugs mined from popular open-source repositories to ensure ecological validity. Task 1 involved a serialization logical error in protobuf-net, while Task 2 focused on an arithmetic overflow in YamlDotNet. Twelve industry professionals participated in a within-subjects study comparing Robin against a baseline (GitHub Copilot Chat Preview).
Key Results and Insights
Robin achieved a 5x improvement in bug resolution compared to the baseline. Qualitatively, participants compared Robin to a "Senior SDE" or pair programmer. Unlike the baseline, which often confused users by giving code upfront, Robin’s step-by-step guidance lowered the barrier to entry for complex tasks.
As shown in the performance data, while users spent more time interacting "with the tool" when using Robin, they spent significantly less time in "prolonged self-effort" (struggling alone). This shift led to more successful outcomes and higher-quality fixes that addressed logical flaws rather than just hiding symptoms. Furthermore, Robin’s suggested follow-up questions served as a form of scaffolding, teaching users how to structure a debugging conversation effectively.
Conclusion
Robin demonstrates that shifting from "eager" to "collaborative" interaction patterns significantly enhances the utility of AI assistants in complex debugging scenarios. By insisting on gathering context before proposing solutions, Robin helps developers cure the "disease" rather than just the symptom. Future work will explore deeper IDE integration, such as the AI performing UI actions like setting breakpoints directly, and expanding these patterns to other complex software engineering tasks like code migration.
Summary updated