Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors

Zhiyu Yang, Shuo Wang, Yukun Yan, Yang Deng

Abstract

LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs' capabilities to autonomously find and fix runtime logical errors in complex data science code remain largely unexplored. To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging. DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. DSDBench includes 1,117 annotated samples with 741 cause-effect error pairs and runtime error messages. Evaluations of state-of-the-art LLMs on DSDBench show significant performance gaps, highlighting challenges in debugging logical runtime errors in data science code. DSDBench offers a crucial resource to evaluate and improve LLMs' debugging and reasoning capabilities, enabling more reliable AI-assisted data science in the future.

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, their ability to debug complex, real-world data science code remains a critical bottleneck. Most existing benchmarks focus on isolated functions with single syntax errors. However, real-world data science is "messy," relying on complex libraries like pandas and scikit-learn where errors are often logical rather than syntactic.

This research identifies two primary challenges: Multi-Hop Errors, where the root cause of a failure is distant from the eventual crash site, and Multi-Bug Errors, where multiple overlapping issues interact. To evaluate these capabilities, the authors introduce DSDBench, a benchmark specifically designed to test the "understanding" gap in AI-assisted coding.

Methodology and Dataset

DSDBench consists of 1,117 meticulously annotated samples derived from established benchmarks like DABench, MatPlotBench, and DSEval. The construction pipeline utilizes a "dual-injection" strategy to create realistic buggy code.

The DSDBench construction pipeline, illustrating the injection of logical errors and the annotation of cause-effect pairs.

As shown in the pipeline above, the authors used GPT-4o for "Strong LLM Injection" to introduce plausible logic faults (e.g., API misuses) and Llama-3.1-8B for "Weak LLM Injection" to generate novice-style mistakes. Ground truth was established using the snoop library to trace execution and map "cause lines" to "effect lines." The dataset covers a wide array of libraries, with pandas being the most prevalent and difficult to debug due to its complex internal states.

Key Insights and Results

The "Generation vs. Understanding" Gap

The study reveals that models excelling at code generation often struggle with debugging. While frontier models can identify where a program crashes (the effect), their accuracy drops significantly when tracing the crash back to its original logical error (the cause).

The Multi-Bug Cliff

Performance degrades sharply as the complexity of the debugging task increases. When multiple bugs exist in a single snippet, they can "mask" one another, making identification significantly harder.

Precision of cause and effect line detection as the number of bugs increases, showing a significant performance drop across all models.

As illustrated in the figure above, cause-line accuracy for top models like GPT-4o and Qwen-72B plummeted as the number of bugs increased. For single-bug cases, top models achieved 44-48% accuracy, but this fell to between 12% and 21% in multi-bug scenarios. Interestingly, "Large Reasoning Models" (LRMs) like DeepSeek-R1 showed better resilience, likely due to their ability to mentally simulate code execution traces.

Conclusion

DSDBench highlights that current LLMs are far from autonomous debuggers in data science contexts. Future work will need to address "silent" functional bugs that do not cause crashes and expand the scope to repository-level debugging. Improving the structured reasoning of models—perhaps through self-debugging loops—remains a prerequisite for reliable AI-assisted data science.

Summary updated

Abstract

LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs' capabilities to autonomously find and fix runtime logical errors in complex data science code remain largely unexplored. To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging. DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. DSDBench includes 1,117 annotated samples with 741 cause-effect error pairs and runtime error messages. Evaluations of state-of-the-art LLMs on DSDBench show significant performance gaps, highlighting challenges in debugging logical runtime errors in data science code. DSDBench offers a crucial resource to evaluate and improve LLMs' debugging and reasoning capabilities, enabling more reliable AI-assisted data science in the future.DSDBench is publicly available at https://github.com/KevinCL16/DSDBench.
BibTeX
@inproceedings{yang2025why,
  author = {Zhiyu Yang and Shuo Wang and Yukun Yan and Yang Deng},
  title = {{Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=ByADwt2aEB},
}
Figure