TL;DR
This paper introduces DEBUGEVAL, a new multi-stage benchmark that mimics the human debugging process (localization, identification, repair, and recognition) to better evaluate LLM coding capabilities. To address the poor performance of 7B-scale models on this benchmark, the authors develop COAST, a multi-agent data synthesis framework that generates high-quality training data, elevating smaller models to debugging performance on par with GPT-3.5.
What Problem Does This Paper Address?
When humans debug code, they rarely just rewrite a broken function from scratch in one shot. Instead, they follow a systematic, multi-stage workflow: locating the buggy line, analyzing the root cause, applying a fix, and verifying the corrected code against the original.
However, existing code debugging benchmarks predominantly focus only on the Code Repair stage. This creates a significant evaluation gap. Large Language Models (LLMs) might be able to patch a simple error by pattern matching, but do they actually understand code semantics and error types? The authors show that smaller, 7B-scale models consistently struggle with holistic debugging, frequently failing to identify or locate errors even when they can occasionally fix them.
Methods
To bridge this gap, the authors propose the COmmunicative Agent-based data SynThesis (COAST) framework, which uses a three-agent cooperative system to generate high-quality Supervised Fine-Tuning (SFT) data:
- Code Quizzer: Utilizing a powerful model (DeepSeek-Coder-V2), this agent generates diverse code debugging problems based on seed examples.
- Code Learner: This agent uses the same backbone as the target model being trained. It attempts to solve the Quizzer's problems and acts as a filter. If the Learner answers a question correctly, the problem is discarded. If it answers incorrectly, the problem is flagged as having high "educational value" for fine-tuning.
- Code Teacher: A strong LLM agent that reviews the failed problems and generates detailed Chain-of-Thought (CoT) explanations and correct answers to guide the Learner.
Crucially, the authors discovered that while CoT reasoning helps significantly with localization and identification tasks, it actually hurts Code Repair performance by introducing structural noise into the code. Therefore, their final NeuDebugger models are trained using a hybrid strategy: CoT data for localization, identification, and recognition, but direct code answers for the repair task.
Datasets
The authors introduced and utilized the following datasets:
- DEBUGEVAL (New Benchmark): Contains 5,712 test instances spanning three programming languages (Python, C++, Java) and 18 distinct error types. The bugs are sourced from both real-world human trials (AtCoder) and GPT-4 generations to simulate realistic software development challenges.
- COAST Synthesized Data: A training dataset consisting of 4,681 Bug Localization, 4,474 Bug Identification, 11,317 Code Repair, and 4,420 Code Recognition instances.
- Vanilla SFT Baselines: For comparison, models were also trained on existing datasets like UltraInteract, InstructCoder, and RepairLlama.
Key Insights and Ideas
- Failure-Driven Learning: Filtering synthetic data based on what the student model fails to understand creates a highly efficient curriculum, outperforming standard human-curated or un-filtered GPT-4 datasets.
- The CoT Trade-Off in Coding: Chain-of-Thought is a double-edged sword. It provides the logical scaffolding needed to classify and locate bugs, but when generating raw code fixes, the extra text can disrupt the model's adherence to strict code syntax.
- Overcoming the "Logical Error" Bias: Untrained 7B models exhibit a strong bias toward classifying almost every bug as a "Logical Error" due to a lack of nuanced understanding. COAST training successfully flattens this distribution, giving models the ability to distinguish syntax and reference errors accurately.
Key Results
- NeuDebugger-DS-6.7B (fine-tuned using COAST) achieved an average debugging accuracy improvement of 27.7% over its baseline model, achieving performance comparable to GPT-3.5-Turbo across the DEBUGEVAL benchmark.
- NeuDebugger-Llama3-8B saw a 4.1% overall improvement, demonstrating that while COAST works universally, code-specialized base models (like DeepSeek) benefit far more from debugging SFT data than general-purpose models.
Limitations and Future Work
While DEBUGEVAL expands evaluation to four distinct tasks, the authors note that incorporating even more real-world debugging scenarios (such as multi-file debugging or runtime log analysis) would make the benchmark more comprehensive. Additionally, the COAST framework's efficacy is fundamentally bounded by the capabilities of its "Teacher" and "Quizzer" models; if the underlying foundation model lacks advanced reasoning, the quality of the synthesized explanations degrades. Future work will explore scaling these agent interactions to even more complex software architectures.
Summary updated