Exploring Interaction Patterns for Debugging: Enhancing Conversational Capabilities of AI-assistants

Bhavya Chopra, Yasharth Bajpai, Param Biyani, Gustavo Soares, Arjun Radhakrishna, Chris Parnin, Sumit GulwaniarXiv 2024 · DOI · arXiv

Abstract

Current AI coding assistants often provide superficial fixes due to a lack of context and a "one-shot" interaction model. This paper introduces Robin, a debugging assistant that utilizes "insert expansion" and guided turn-taking to engage in multi-turn collaborative troubleshooting. Evaluated through a user study with 12 industry professionals, Robin demonstrated a 5x improvement in bug resolution rates and a 2x improvement in bug localization compared to standard chat-based baselines by shifting the interaction from immediate answers to investigative dialogue.

Introduction

While Large Language Models (LLMs) are now integrated into most IDEs, their interaction model remains largely stuck in a "unilateral question-answer" loop. The developer asks a question, and the AI attempts to provide a final answer immediately. This approach leads to two major issues: pre-mature bug localization, where AIs provide "band-aid" fixes for symptoms rather than root causes, and high cognitive load, placing the burden of investigation entirely on the developer. The researchers aim to align AI assistants with Gricean maxims of conversation—specifically quality (avoiding speculation) and quantity (providing the right amount of info at the right time).

Methodology

Robin is an extension for Visual Studio that implements a specialized agent-based workflow to move beyond one-shot responses. The system architecture consists of several key components designed to foster collaboration:

  • Hardness Classifier: Analyzes the bug and context to decide if it's a "one-shot" fix (Eager) or requires a "collaborative" approach.
  • Insert Expansion Pattern: Instead of answering immediately, Robin initiates auxiliary turns to ask the user for missing information, such as variable values or specific method code.
  • Collaborative Responder: Guides the user through debugging workflows, suggesting specific places to set breakpoints or variables to watch.
  • Follow-up Generator: Creates context-aware prompts that act as a style guide for the user, making it easier to continue the investigative dialogue.
The Robin workflow architecture, featuring the Hardness Classifier and specialized responders.

Evaluation

The system was evaluated using real-world bugs mined from popular open-source repositories to ensure ecological validity. Task 1 involved a serialization logical error in protobuf-net, while Task 2 focused on an arithmetic overflow in YamlDotNet. Twelve industry professionals participated in a within-subjects study comparing Robin against a baseline (GitHub Copilot Chat Preview).

Key Results and Insights

Robin achieved a 5x improvement in bug resolution compared to the baseline. Qualitatively, participants compared Robin to a "Senior SDE" or pair programmer. Unlike the baseline, which often confused users by giving code upfront, Robin’s step-by-step guidance lowered the barrier to entry for complex tasks.

Time spent on observed activities across Task 1 and Task 2, comparing Baseline and Robin.

As shown in the performance data, while users spent more time interacting "with the tool" when using Robin, they spent significantly less time in "prolonged self-effort" (struggling alone). This shift led to more successful outcomes and higher-quality fixes that addressed logical flaws rather than just hiding symptoms. Furthermore, Robin’s suggested follow-up questions served as a form of scaffolding, teaching users how to structure a debugging conversation effectively.

Conclusion

Robin demonstrates that shifting from "eager" to "collaborative" interaction patterns significantly enhances the utility of AI assistants in complex debugging scenarios. By insisting on gathering context before proposing solutions, Robin helps developers cure the "disease" rather than just the symptom. Future work will explore deeper IDE integration, such as the AI performing UI actions like setting breakpoints directly, and expanding these patterns to other complex software engineering tasks like code migration.

Summary updated

Abstract

The widespread availability of Large Language Models (LLMs) within Integrated Development Environments (IDEs) has led to their speedy adoption. Conversational interactions with LLMs enable programmers to obtain natural language explanations for various software development tasks. However, LLMs often leap to action without sufficient context, giving rise to implicit assumptions and inaccurate responses. Conversations between developers and LLMs are primarily structured as question-answer pairs, where the developer is responsible for asking the the right questions and sustaining conversations across multiple turns. In this paper, we draw inspiration from interaction patterns and conversation analysis -- to design Robin, an enhanced conversational AI-assistant for debugging. Through a within-subjects user study with 12 industry professionals, we find that equipping the LLM to -- (1) leverage the insert expansion interaction pattern, (2) facilitate turn-taking, and (3) utilize debugging workflows -- leads to lowered conversation barriers, effective fault localization, and 5x improvement in bug resolution rates.
BibTeX
@misc{chopra2024exploring,
  author = {Bhavya Chopra and Yasharth Bajpai and Param Biyani and Gustavo Soares and Arjun Radhakrishna and Chris Parnin and Sumit Gulwani},
  title = {{Exploring Interaction Patterns for Debugging: Enhancing Conversational Capabilities of AI-assistants}},
  year = {2024},
  eprint = {2402.06229},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.06229},
}
Figure