Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebooks

Konstantin Grotov, Artem Borzilov, Maksim Krivobok, Timofey Bryksin, Yaroslav Zharov

TL;DR

This paper introduces an AI agent designed to automatically resolve errors in computational notebooks, a common but often bug-prone development environment. While the agent demonstrates strong error resolution capabilities, user feedback highlights challenges with its interface and control.

What Problem Does This Paper Address?

Computational notebooks, like Jupyter, have become essential for data science and research due to their interactive and flexible nature. However, their stateful and non-linear structure makes them prone to errors and reproducibility issues. Existing AI tools, often trained on linear scripts, struggle to effectively debug these complex notebooks. This paper tackles the challenge of creating an AI agent that can autonomously navigate and fix errors within the unique environment of a computational notebook.

Methods

The core of the proposed solution is an AI agent that interacts with a computational notebook environment. This agent is built upon a Large Language Model (LLM), specifically GPT-4, chosen for its function-calling capabilities. The system comprises three main parts:

  • Agent: Orchestrates communication between the LLM and the notebook, manages prompts, and translates LLM decisions into actions.
  • Environment: The computational notebook itself (integrated into JetBrains Datalore), responsible for executing agent-provided actions and returning observations.
  • User Interface: A panel within Datalore that displays the agent's actions and explanations, allowing users to initiate the process.

The agent employs a "reflection" strategy, where it prompts the LLM to consider previous actions' outcomes before deciding on the next step. It's equipped with tools to create, edit, and execute cells, and a "Finish" action to signal completion. The agent's goal is to resolve runtime exceptions by making necessary code changes or cell executions, avoiding "hacks" like simply deleting problematic code.

Datasets

The paper mentions using a dataset of fine-grained Jupyter Notebook execution logs for evaluation. This dataset, collected during a hackathon, contains over 100 hours of logs from 20 participants, capturing cell additions, executions, and deletions. This dataset is described as being part of another paper under review and is available upon request.

Key Insights and Ideas

  • Notebooks are a unique challenge: The stateful and non-linear nature of notebooks presents a distinct debugging problem compared to traditional scripts, requiring agents that can interact dynamically with the environment.
  • Agentic approach is promising: LLM-powered agents, capable of iterative interaction and tool use, show potential for tackling complex, state-dependent tasks like notebook debugging.
  • User control is crucial: While the AI agent excels at finding solutions, users expressed a strong desire for more control over the agent's actions and better transparency into its process.
  • Cost vs. Capability Trade-off: The agentic system incurs higher token costs due to its iterative nature and growing context, but this is offset by its effectiveness in resolving errors.

Key Results

  • Error Resolution: The AI agent successfully resolved errors in most cases, often within the first or second step. Users rated the agent's error resolution capabilities higher than a single-action solution.
  • Cost Analysis: The agentic system consumed approximately three times more input tokens than the single-action solution but a similar amount of response tokens. The average cost per error resolution was $0.22 for the AI agent versus $0.09 for the single-action solution.
  • User Study:
    • Error Resolution Capability: Users rated the AI agent's error resolution capabilities higher (mean 4.03 ± 0.31) compared to the single-step solution (mean 3.41 ± 0.49).
  • Productivity: Both systems received similar high ratings for positive productivity feedback (AI agent: 4.08 ± 0.43; single-step: 4.10 ± 0.35). However, users found interacting with the agent mentally more taxing.
  • User Experience: The AI agent received a worse rating for negative productivity feedback (mean 1.57 ± 0.18) compared to the single-step solution (mean 1.31 ± 0.09), attributed to an overloaded UI and lack of user control.
  • System Usability Scale (SUS): Both systems scored similarly well (AI agent: 72.7; single-step: 72.8), indicating good usability.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • User Interface (UI) and Control: The primary limitation identified is the UI, which users found overwhelming and lacking sufficient control. Future work should focus on improving transparency, visualization of agent actions, and allowing users to specify edits more granularly.
  • Cost Mitigation: While the cost is deemed acceptable, future work could explore using smaller, cheaper LLMs and implementing context caching techniques to reduce token consumption.
  • Quantitative Accuracy Evaluation: The paper did not conduct a quantitative evaluation of the accuracy of the agent's solutions, focusing instead on whether the code executed without errors. Developing metrics to assess the correctness and relevance of solutions is a future research area.
  • Security: The lack of a secure sandbox environment for the agent to explore is a critical limitation, as it raises concerns about data and code safety.
  • Meaningful Solutions: The paper notes the challenge of distinguishing between true problem resolution and "hallucinations" or superficial fixes. Future research should focus on developing methods to automatically assess the quality and root-cause addressing nature of agent solutions.

Summary updated

Abstract

Computational notebooks became indispensable tools for research-related development, offering unprecedented interactivity and flexibility in the development process. However, these benefits come at the cost of reproducibility and an increased potential for bugs.With the rise of code-fluent Large Language Models empowered with agentic techniques, smart bug-fixing tools with a high level of autonomy have emerged.However, those tools are tuned for classical script programming and still struggle with non-linear computational notebooks.In this paper, we present an AI agent designed specifically for error resolution in a computational notebook. We have developed an agentic system capable of exploring a notebook environment by interacting with it—similar to how a user would—and integrated the system into the JetBrains service for collaborative data science called Datalore.We evaluate our approach against the pre-existing single-action solution by comparing costs and conducting a user study. Users rate the error resolution capabilities of the agentic system higher but experience difficulties with UI. We share the results of the study and consider them valuable for further improving user-agent collaboration.
BibTeX
@inproceedings{grotov2024debug,
  author = {Konstantin Grotov and Artem Borzilov and Maksim Krivobok and Timofey Bryksin and Yaroslav Zharov},
  title = {{Debug Smarter, Not Harder: AI Agents for Error Resolution in Computational Notebooks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=L2rZIaGyov},
}
Figure