Enhancing the Code Debugging Ability of LLMs via Communicative Agent Based Data Refinement

Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, Ge Yu

Abstract

While Large Language Models (LLMs) have demonstrated remarkable proficiency in code generation, their ability to effectively debug code remains a significant challenge, particularly for smaller-scale models. This paper introduces DEBUGEVAL, a novel benchmark that treats code debugging as a multi-stage, human-like process involving localization, identification, repair, and recognition. To bridge the performance gap in 7B-scale models, the authors propose COAST (COmmunicative Agent based data SynThesis), a multi-agent framework where specialized agents collaborate to generate high-quality training data by identifying and explaining model failures. Models fine-tuned using this approach, such as NeuDebugger, achieve performance parity with GPT-3.5-Turbo, highlighting the potential of communicative data refinement for specialized reasoning tasks.

Introduction

The rapid advancement of Large Language Models (LLMs) has revolutionized automated software engineering. However, debugging—a core component of the development cycle—remains a bottleneck. Most existing evaluations simplify debugging to a single "Code Repair" task, overlooking the nuanced cognitive workflow humans employ. Human debugging is iterative and multi-faceted, requiring the developer to first locate the error, identify its nature, formulate a fix, and finally verify the correction.

Smaller models (e.g., 7B parameters) often struggle with these intermediate steps, frequently failing to grasp the underlying semantics of a bug. This "7B Gap" is characterized by a lack of nuance; for instance, base models often default to labeling any error as a "Logical Error" without further distinction. To address this, high-quality, multi-stage training data is required to teach models the full debugging lifecycle.

The DEBUGEVAL Benchmark

To provide a more comprehensive evaluation of debugging capabilities, the authors developed DEBUGEVAL. This benchmark consists of 5,712 instances across three programming languages (Python, C++, Java) and 18 error types. Unlike traditional benchmarks, DEBUGEVAL decomposes the debugging process into four distinct tasks:

  1. Localization: Identifying the specific code fragment containing the bug.
  2. Identification: Classifying the error type (e.g., Syntax, Reference, Logical).
  3. Repair: Providing the corrected code solution.
  4. Recognition: Distinguishing between buggy and correct code versions.
The DEBUGEVAL benchmark decomposes debugging into four distinct tasks: Localization, Identification, Repair, and Recognition.

The COAST Framework

The core contribution of this work is the COmmunicative Agent based data SynThesis (COAST) framework. COAST moves beyond simple data scraping by employing a "society" of three specialized LLM agents to synthesize a high-quality curriculum for Supervised Fine-Tuning (SFT).

  • Code Quizzer: Utilizing a high-end model (e.g., GPT-4), this agent generates diverse debugging problems based on seed examples from DEBUGEVAL.
  • Code Learner: This is the target model (e.g., a 7B-scale model) undergoing training. It attempts to solve the problems posed by the Quizzer. Problems that the Learner fails to solve are flagged as having high "educational value."
  • Code Teacher: For every failure identified, the Teacher agent provides a detailed, step-by-step Chain-of-Thought (CoT) explanation and the correct solution.
The COAST framework utilizes a multi-agent system (Quizzer, Learner, Teacher) to synthesize high-quality debugging training data through a "quiz-fail-teach" loop.

By focusing on the specific weaknesses of the Learner, COAST creates a tailored dataset that emphasizes "learning from failure." The final model, NeuDebugger, is fine-tuned on these curated lessons.

Key Insights and Results

The evaluation of NeuDebugger reveals several critical insights into LLM debugging:

  • Performance Parity: NeuDebugger-DS-6.7B and NeuDebugger-Llama3-8B achieved overall scores (54-56%) comparable to GPT-3.5-Turbo (55.1%). On the DEBUGEVAL benchmark, NeuDebugger-DS-6.7B showed a 27.7% improvement over its base version.
  • The Utility of Failure: Training on COAST-generated data, which focuses on model failures, outperformed training on standard human-curated or raw GPT-4-generated datasets.
  • CoT Nuance: While Chain-of-Thought reasoning is highly effective for localization and identification, it can occasionally introduce noise during the "Code Repair" task. Consequently, NeuDebugger is optimized to use CoT for analysis but provides direct answers for final repairs.
  • Addressing the 7B Gap: Before fine-tuning, smaller models showed a massive bias toward "Logical Error" classifications (up to 98% in some tests). NeuDebugger significantly improved the model's ability to distinguish between different error types.

Conclusion and Future Work

The COAST framework and DEBUGEVAL benchmark represent a significant step toward human-like debugging in LLMs. By decomposing the task and focusing on communicative data refinement, smaller models can achieve performance previously reserved for much larger systems. Future research will focus on expanding these methods to complex, multi-file software projects and integrating real-world execution feedback from compilers and test suites directly into the agent loop.

Summary updated

Abstract

Debugging is a vital aspect of software development, yet the debugging capabilities of Large Language Models (LLMs) remain largely unexplored. This paper first introduces DEBUGEVAL, a comprehensive benchmark designed to evaluate the debugging capabilities of LLMs. DEBUGEVAL collects data from existing high-quality datasets and designs four different tasks to evaluate the debugging effectiveness, including BUG Localization, BUG Identification, Code Review, and Code Repair. Additionally, to enhance the code debugging ability of LLMs, this paper proposes a CoMmunicative Agent BaSed DaTa REfinement FRamework (MASTER), which generates the refined code debugging data for supervised finetuning. Specifically, MASTER employs the Code Quizzer to generate refined data according to the defined tasks of DEBUGEVAL. Then the Code Learner acts as a critic and reserves the generated problems that it can not solve. Finally, the Code Teacher provides a detailed Chain-of-Thought based solution to deal with the generated problem. We collect the synthesized data and finetune the Code Learner to enhance the debugging ability and conduct the NeuDebugger model. Our experiments evaluate various LLMs and NeuDebugger in the zero-shot setting on DEBUGEVAL. Experimental results demonstrate that these 7B-scale LLMs have weaker debugging capabilities, even these code-oriented LLMs. On the contrary, these larger models (over 70B) show convincing debugging ability. Our further analyses illustrate that MASTER is an effective method to enhance the code debugging ability by synthesizing data for Supervised Fine-Tuning (SFT) LLMs.
BibTeX
@inproceedings{yang2025enhancing,
  author = {Weiqing Yang and Hanbin Wang and Zhenghao Liu and Xinze Li and Yukun Yan and Shuo Wang and Yu Gu and Minghe Yu and Zhiyuan Liu and Ge Yu},
  title = {{Enhancing the Code Debugging Ability of LLMs via Communicative Agent Based Data Refinement}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.139},
}
Figure