Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors

Zhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng

TL;DR

This paper introduces DSDBench, the first benchmark specifically designed to evaluate Large Language Models (LLMs) on the complex task of debugging data science code. It highlights that current LLM benchmarks often overlook real-world debugging challenges like multi-hop errors (where the cause is distant from the symptom) and multi-bug scenarios, revealing significant performance gaps in state-of-the-art LLMs.

What Problem Does This Paper Address?

The rapid advancement of LLMs has led to impressive capabilities in code generation and even simple bug fixing. However, most existing benchmarks for evaluating LLMs in code repair focus on isolated, single errors that are syntactically or functionally incorrect. This is a far cry from the reality of data science debugging, which often involves:

  • Multi-hop errors: A small mistake early in a complex data pipeline can manifest as a crash much later, making it difficult to trace the root cause.
  • Multi-bug scenarios: Data science scripts frequently contain multiple logical errors that interact and complicate the debugging process.
  • Black-box libraries and exploratory code: Data scientists heavily rely on libraries like pandas and scikit-learn, and often write code in interactive environments (like Jupyter notebooks) that are prone to subtle, data-dependent logical errors.

Current benchmarks fail to capture these nuances, leaving a critical gap in understanding LLMs' true debugging capabilities in practical, real-world data science contexts.

Methods

To address this gap, the authors developed DSDBench, a novel benchmark and dataset. The construction process involved several key stages:

  1. Data Sourcing: They leveraged existing data science coding benchmarks like DABench, MatPlotBench, and DSEval to ensure realistic tasks and library coverage (pandas, NumPy, scikit-learn, etc.).
  2. Correct Code Preparation: For each benchmark task, they first generated or extracted correct, error-free Python code. This involved using an agent-based annotation framework for DABench and manual verification for MatPlotBench.
  3. Automated Error Injection: This was a crucial step. They employed two methods:
  4. Strong LLM-based Injection: Using GPT-40, they injected plausible runtime errors into correct code by modifying API calls, parameter settings, or data shapes. This method was designed to create realistic, often multi-hop errors.
  5. Weak LLM-based Direct Error Generation: Using a less capable model (Llama-3.1-8B), they generated code from scratch, which naturally introduced beginner-style mistakes and multi-hop errors.
  6. Error Annotation: For each buggy snippet, they meticulously annotated three ground truths:
  7. cause_error_line: The exact line where the logical error originates.
  8. effect_error_line: The exact line where the interpreter reports the error (the symptom).
  9. runtime_error_message: The exact error message produced by the Python interpreter.
  10. Quality Assurance: A two-stage verification process, involving code checks and LLM-assisted review, ensured the accuracy and reliability of the annotations.

Datasets

The primary dataset introduced is DSDBench, comprising 1,117 meticulously annotated samples. These samples are derived from existing data science benchmarks (DABench, MatPlotBench, DSEval) and feature automatically synthesized multi-hop and multi-bug code snippets. The dataset includes 741 multi-error examples and 376 single-error examples, with detailed cause-effect error line pairs and runtime error messages.

Key Insights and Ideas

  • The "Generation vs. Understanding" Gap: The evaluations revealed a significant performance drop for LLMs when tasked with debugging complex, multi-hop, or multi-bug errors, even for models that excel at code generation. This highlights a disconnect between an LLM's ability to produce code and its ability to truly understand and reason about its execution.
  • Multi-Hop Errors are a Major Hurdle: LLMs struggle to trace errors back to their root cause when the cause line differs significantly from the effect line. This indicates a deficiency in their ability to simulate code execution flow and identify distant causal relationships.
  • Multi-Bug Debugging is Extremely Challenging: Performance plummets dramatically in multi-bug scenarios, with models failing to identify all constituent bugs. This suggests that current LLMs are not adept at handling concurrent logical errors.
  • Large Reasoning Models (LRMs) Show Promise: Models categorized as LRMs (like Gemini 2.0 Flash Thinking and DeepSeek-R1) generally outperformed standard LLMs, particularly in multi-bug scenarios, suggesting that their enhanced reasoning capabilities are crucial for complex debugging.
  • Self-Refinement Helps, But Isn't a Silver Bullet: While prompting strategies like self-refinement can improve performance, they don't fully bridge the gap, underscoring the inherent difficulty of the DSDBench tasks.

Key Results

  • Significant Performance Gaps: State-of-the-art LLMs, including GPT-40 and Claude 3.5 Sonnet, showed accuracy rates around 40-48% for single-bug cause line detection but dropped to as low as 10-20% for multi-bug cause line detection.
  • Effect Line Accuracy is Lower: LLMs consistently performed worse at identifying the effect line (where the error manifests) compared to the cause line, indicating difficulty in tracing execution flow.
  • Multi-Bug Accuracy Drops Sharply: For multi-bug scenarios, accuracy for cause line detection dropped to around 20% for top models on the full dataset.
  • LRMs Outperform LLMs: On a subset of DSDBench, LRMs achieved significantly higher accuracy in multi-bug scenarios, with some models reaching over 30% for cause line detection, compared to around 10-20% for standard LLMs.
  • Error Type and Library Impact: Performance varied by error type and library. Pandas proved particularly difficult to debug.

Limitations and Future Work

  • Focus on Python and Interpreter Errors: The current benchmark is limited to Python and interpreter-visible (crashing) bugs. Future work could expand to other languages and include functional bugs that don't necessarily cause a crash.
  • Repository-Level Debugging: The benchmark currently focuses on single-file scripts. Extending it to multi-file dependencies and repository-level debugging would be a valuable next step.
  • Data Science Domain Specificity: While DSDBench is tailored for data science, future work could explore LLM debugging capabilities in other complex coding domains.
  • Scalability of Annotation: While automated error injection was used, manual verification was still necessary, highlighting the ongoing challenge of creating large-scale, high-quality debugging datasets.

Summary updated

Abstract

LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs’ capabilities to autonomously find and fix runtime logical errors in complex data science code remain largely unexplored. To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging. DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. DSDBench includes 1,117 annotated samples with 741 cause-effect error pairs and runtime error messages. Evaluations of state-of-the-art LLMs on DSDBench show significant performance gaps, highlighting challenges in debugging logical runtime errors in data science code. DSDBench offers a crucial resource to evaluate and improve LLMs’ debugging and reasoning capabilities, enabling more reliable AI-assisted data science in the future.
BibTeX
@inproceedings{yang2025why,
  author = {Zhiyu Yang and Shuo Wang and Yukun Yan and Yang Deng},
  title = {{Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=ByADwt2aEB},
}
Figure