Abstract
While Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, their ability to effectively debug code remains a significant challenge, particularly for smaller-scale models. This paper introduces DEBUGEVAL, a novel benchmark that treats code debugging as a multi-stage, human-like process involving localization, identification, repair, and recognition. To bridge the performance gap in 7B-scale models, the authors propose COAST (COmmunicative Agent based data SynThesis), a multi-agent framework where specialized agents collaborate to generate high-quality training data by identifying and explaining model failures. Models fine-tuned using this approach, such as NeuDebugger, achieve performance parity with GPT-3.5-Turbo, highlighting the potential of communicative data refinement for specialized reasoning tasks.
Introduction
The rapid advancement of Large Language Models (LLMs) has revolutionized automated software engineering. However, debugging—a core component of the development cycle—remains a bottleneck. Most existing evaluations simplify debugging to a single "Code Repair" task, overlooking the nuanced cognitive workflow humans employ. Human debugging is iterative and multi-faceted, requiring the developer to first locate the error, identify its nature, formulate a fix, and finally verify the correction.
Smaller models (e.g., 7B parameters) often struggle with these intermediate steps, frequently failing to grasp the underlying semantics of a bug. This "7B Gap" is characterized by a lack of nuance; for instance, base models often default to labeling any error as a "Logical Error" without further distinction. To address this, high-quality, multi-stage training data is required to teach models the full debugging lifecycle.
The DEBUGEVAL Benchmark
To provide a more comprehensive evaluation of debugging capabilities, the authors developed DEBUGEVAL. This benchmark consists of 5,712 instances across three programming languages (Python, C++, Java) and 18 error types. Unlike traditional benchmarks, DEBUGEVAL decomposes the debugging process into four distinct tasks:
- Localization: Identifying the specific code fragment containing the bug.
- Identification: Classifying the error type (e.g., Syntax, Reference, Logical).
- Repair: Providing the corrected code solution.
- Recognition: Distinguishing between buggy and correct code versions.
The COAST Framework
The core contribution of this work is the COmmunicative Agent based data SynThesis (COAST) framework. COAST moves beyond simple data scraping by employing a "society" of three specialized LLM agents to synthesize a high-quality curriculum for Supervised Fine-Tuning (SFT).
- Code Quizzer: Utilizing a high-end model (e.g., GPT-4), this agent generates diverse debugging problems based on seed examples from DEBUGEVAL.
- Code Learner: This is the target model (e.g., a 7B-scale model) undergoing training. It attempts to solve the problems posed by the Quizzer. Problems that the Learner fails to solve are flagged as having high "educational value."
- Code Teacher: For every failure identified, the Teacher agent provides a detailed, step-by-step Chain-of-Thought (CoT) explanation and the correct solution.
By focusing on the specific weaknesses of the Learner, COAST creates a tailored dataset that emphasizes "learning from failure." The final model, NeuDebugger, is fine-tuned on these curated lessons.
Key Insights and Results
The evaluation of NeuDebugger reveals several critical insights into LLM debugging:
- Performance Parity: NeuDebugger-DS-6.7B and NeuDebugger-Llama3-8B achieved overall scores (54-56%) comparable to GPT-3.5-Turbo (55.1%). On the DEBUGEVAL benchmark, NeuDebugger-DS-6.7B showed a 27.7% improvement over its base version.
- The Utility of Failure: Training on COAST-generated data, which focuses on model failures, outperformed training on standard human-curated or raw GPT-4-generated datasets.
- CoT Nuance: While Chain-of-Thought reasoning is highly effective for localization and identification, it can occasionally introduce noise during the "Code Repair" task. Consequently, NeuDebugger is optimized to use CoT for analysis but provides direct answers for final repairs.
- Addressing the 7B Gap: Before fine-tuning, smaller models showed a massive bias toward "Logical Error" classifications (up to 98% in some tests). NeuDebugger significantly improved the model's ability to distinguish between different error types.
Conclusion and Future Work
The COAST framework and DEBUGEVAL benchmark represent a significant step toward human-like debugging in LLMs. By decomposing the task and focusing on communicative data refinement, smaller models can achieve performance previously reserved for much larger systems. Future research will focus on expanding these methods to complex, multi-file software projects and integrating real-world execution feedback from compilers and test suites directly into the agent loop.
Summary updated