debug-gym: A Text-Based Environment for Interactive Debugging

TL;DR

This paper introduces debug-gym, a novel text-based environment that allows AI agents to interactively debug code using tools like Python's pdb. This aims to bridge the gap between current LLM coding assistants, which often struggle with complex bugs, and human developers who leverage interactive debugging for deeper insights.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used for code generation, but they often falter when faced with complex bugs, especially when the problem deviates from their training data. Current approaches primarily rely on execution feedback (error messages) to guide code repair. However, this can be insufficient for intricate bugs where understanding program state, variable values, and execution flow is crucial. Human developers overcome this by using interactive debugging tools like pdb to step through code, inspect variables, and gain a deeper understanding of the bug's root cause. This paper addresses the challenge of enabling LLM-based agents to perform similar interactive debugging, thereby improving their code-repairing capabilities.

Methods

The core contribution is debug-gym, a Python library that provides a text-based environment for interactive debugging. It simulates a coding environment where an agent can:

• Interact with a codebase: Agents can access and modify files within a repository.
• Utilize a toolbox: A set of pre-defined tools, including a Python debugger (pdb), a file viewer (view), and a code editor (rewrite), are available.
• Receive structured observations: The environment provides observations in a structured text format (e.g., JSON), including code snippets, error messages, and tool outputs.
• Take text-based actions: Agents issue commands to interact with the environment and its tools.

debug-gym is designed to be extensible, allowing users to add custom tools. It also incorporates safety features like Docker containerization to isolate the execution environment. The environment formalizes the debugging process as a Partially Observable Markov Decision Process (POMDP).

Datasets

The paper evaluates debug-gym using three coding benchmarks:

• Aider: A collection of 133 Python programming exercises from Exercism, focusing on simple function-level code generation.
• Mini-nightmare: 10 hand-crafted buggy Python code examples (average 40 lines) designed to highlight scenarios where interactive debugging is particularly useful.
• SWE-bench-Lite: A curated subset of 300 real-world coding problems from the SWE-bench benchmark, requiring understanding of larger codebases and component interactions.

Key Insights and Ideas

• Interactive debugging is crucial for complex bugs: The results strongly suggest that while LLMs can perform basic code generation, interactive debugging tools are essential for tackling more challenging bugs that require understanding runtime state.
• Tool access matters, but effective usage is key: Simply providing LLMs with debugging tools doesn't guarantee proficiency. The agents' ability to effectively leverage these tools is critical, and this often depends on the LLM's backbone and prompt engineering.
• Stronger LLMs benefit more from interactive tools: While weaker LLMs may struggle to utilize debugging tools effectively, stronger models (like Claude 3.7 Sonnet) show a more diverse and strategic use of tools, leading to better performance.
• "Curiosity-driven exploration" can be beneficial: The success of stronger models like Claude 3.7 Sonnet on complex tasks like SWE-bench is attributed, in part, to their tendency to explore different tools, even if not immediately obvious how they'll be useful.
• Data scarcity for sequential decision-making is a bottleneck: The authors hypothesize that the limited availability of data representing sequential decision-making processes (like debugging traces) in LLM training corpora hinders their ability to become proficient interactive debuggers.

Key Results

• Aider Benchmark: For simple code generation, adding pdb access did not significantly improve performance for most agents, suggesting that the task did not require deep runtime inspection.
• Mini-nightmare Benchmark: Agents equipped with pdb (debug and debug(5) agents) often outperformed the baseline rewrite agent, especially with stronger LLM backbones. This indicates that interactive debugging tools were helpful in these scenarios.
• SWE-bench-Lite Benchmark:
  • Stronger LLMs (e.g., Claude 3.7 Sonnet) showed significant benefits from interactive debugging tools, achieving higher success rates.
• The debug(5) agent (where pdb is available after the 5th rewrite attempt) consistently outperformed the rewrite agent and, in some cases, even the debug agent, suggesting a strategic approach to tool adoption.
• Robustness scores showed that equipping agents with new tools (like pdb) was less likely to disrupt their existing problem-solving capabilities when using stronger LLM backbones.

Limitations and Future Work

• Agent Design: The current agents have minimal architectural design and prompt engineering, serving primarily as demonstrations of debug-gym's API. Future work will focus on more sophisticated agent architectures and prompt design to guide tool usage.
• Data Scarcity: The lack of sequential decision-making data for training LLMs in interactive debugging is a significant limitation. The authors propose collecting debugging logs from human developers or using strong LLMs to generate and filter trajectories for fine-tuning.
• Trustworthiness: Agents might exploit test cases by finding superficial fixes rather than genuinely solving the problem. Future work includes introducing hidden test cases and dynamically assessing responses.
• Evaluation Metrics: Success rate alone doesn't capture solution efficiency (time/space complexity). Future work may incorporate metrics for solution quality.
• Non-linear Computation Flow: The current environment assumes linear tool calls. Future work could explore DAG-based execution for more complex agent systems.
• Beyond Python: The current environment is Python-specific. Extending it to other languages or leveraging IDE integration protocols is a future direction.
• Test Generation: Integrating on-the-fly test generation with interactive debugging is a promising avenue.