Abstract
LLMs are transforming software development, yet current code generation and code repair benchmarks mainly assess syntactic and functional correctness in simple, single-error cases. LLMs' capabilities to autonomously find and fix runtime logical errors in complex data science code remain largely unexplored. To address this gap, we introduce DSDBench: the Data Science Debugging Benchmark, the first benchmark for systematic evaluation of LLMs on multi-hop error tracing and multi-bug detection in data science code debugging. DSDBench adapts datasets from existing data science task benchmarks, such as DABench and MatPlotBench, featuring realistic data science debugging tasks with automatically synthesized multi-hop, multi-bug code snippets. DSDBench includes 1,117 annotated samples with 741 cause-effect error pairs and runtime error messages. Evaluations of state-of-the-art LLMs on DSDBench show significant performance gaps, highlighting challenges in debugging logical runtime errors in data science code. DSDBench offers a crucial resource to evaluate and improve LLMs' debugging and reasoning capabilities, enabling more reliable AI-assisted data science in the future.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, their ability to debug complex, real-world data science code remains a critical bottleneck. Most existing benchmarks focus on isolated functions with single syntax errors. However, real-world data science is "messy," relying on complex libraries like pandas and scikit-learn where errors are often logical rather than syntactic.
This research identifies two primary challenges: Multi-Hop Errors, where the root cause of a failure is distant from the eventual crash site, and Multi-Bug Errors, where multiple overlapping issues interact. To evaluate these capabilities, the authors introduce DSDBench, a benchmark specifically designed to test the "understanding" gap in AI-assisted coding.
Methodology and Dataset
DSDBench consists of 1,117 meticulously annotated samples derived from established benchmarks like DABench, MatPlotBench, and DSEval. The construction pipeline utilizes a "dual-injection" strategy to create realistic buggy code.
As shown in the pipeline above, the authors used GPT-4o for "Strong LLM Injection" to introduce plausible logic faults (e.g., API misuses) and Llama-3.1-8B for "Weak LLM Injection" to generate novice-style mistakes. Ground truth was established using the snoop library to trace execution and map "cause lines" to "effect lines." The dataset covers a wide array of libraries, with pandas being the most prevalent and difficult to debug due to its complex internal states.
Key Insights and Results
The "Generation vs. Understanding" Gap
The study reveals that models excelling at code generation often struggle with debugging. While frontier models can identify where a program crashes (the effect), their accuracy drops significantly when tracing the crash back to its original logical error (the cause).
The Multi-Bug Cliff
Performance degrades sharply as the complexity of the debugging task increases. When multiple bugs exist in a single snippet, they can "mask" one another, making identification significantly harder.
As illustrated in the figure above, cause-line accuracy for top models like GPT-4o and Qwen-72B plummeted as the number of bugs increased. For single-bug cases, top models achieved 44-48% accuracy, but this fell to between 12% and 21% in multi-bug scenarios. Interestingly, "Large Reasoning Models" (LRMs) like DeepSeek-R1 showed better resilience, likely due to their ability to mentally simulate code execution traces.
Conclusion
DSDBench highlights that current LLMs are far from autonomous debuggers in data science contexts. Future work will need to address "silent" functional bugs that do not cause crashes and expand the scope to repository-level debugging. Improving the structured reasoning of models—perhaps through self-debugging loops—remains a prerequisite for reliable AI-assisted data science.
Summary updated