COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis

Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, Ge Yu

TL;DR

This paper introduces DEBUGEVAL, a new benchmark that evaluates Large Language Models (LLMs) on the full, multi-stage human code debugging process. To address the limitations of smaller LLMs, they propose COAST, a multi-agent framework that synthesizes high-quality training data, enabling 7B-scale LLMs to achieve performance comparable to larger models like GPT-3.5.

What Problem Does This Paper Address?

Code debugging is a critical step in software development, ensuring the reliability of code generated by LLMs. While LLMs are increasingly used for code generation, their ability to debug code is less understood. Existing benchmarks primarily focus on the "Code Repair" stage, neglecting other crucial human debugging steps like "Bug Localization," "Bug Identification," and "Code Recognition." This narrow focus provides an incomplete picture of an LLM's debugging capabilities. Furthermore, smaller LLMs (e.g., 7B parameters) often struggle with complex code semantics, underperforming compared to their larger counterparts. This gap highlights the need for better evaluation methods and improved training strategies for smaller, more accessible LLMs.

Methods

The core contributions of this paper are twofold:

  1. DEBUGEVAL Benchmark: The authors introduce DEBUGEVAL, a comprehensive benchmark designed to mimic the multi-stage human debugging process. It includes four tasks:
  2. Bug Localization: Identifying the specific line of code containing a bug.
  3. Bug Identification: Classifying the type of bug (e.g., Syntax Error, Logical Error).
  4. Code Repair: Generating a corrected version of the buggy code.
  5. Code Recognition: Distinguishing between buggy and correct code snippets. DEBUGEVAL spans multiple programming languages (Python, C++, Java) and uses bugs from both human and GPT-4 generated code.

  6. COAST Framework: To improve the debugging performance of smaller LLMs, the authors propose the COmmunicative Agent-based data SynThesis (COAST) framework. This framework uses a multi-agent system to generate high-quality training data for Supervised Fine-Tuning (SFT). The three agents are:

  7. Code Quizzer: Generates diverse code debugging problems.
  8. Code Learner: Attempts to solve these problems and acts as a critic, flagging incorrect solutions for further processing.
  9. Code Teacher: Provides detailed Chain-of-Thought (CoT) explanations for the identified errors and solutions. This synthesized data is then used to fine-tune the Code Learner, creating a "NeuDebugger" model.

Datasets

  • DEBUGEVAL: A newly introduced benchmark for evaluating LLM debugging capabilities across four tasks and multiple programming languages. It incorporates data from existing benchmarks like DebugBench and LiveCodeBench, as well as human-contributed code from platforms like AtCoder.
  • SFT Datasets: For baseline SFT, the authors use datasets like UltraInteract, InstructCoder, and RepairLlama, which are typically generated by GPT-4 or manually annotated. The COAST framework generates its own SFT data specifically for the DEBUGEVAL tasks.

Key Insights and Ideas

  • Human-like debugging is multi-stage: LLMs need to be evaluated on more than just code repair; localization, identification, and recognition are crucial components of effective debugging.
  • Model scale matters, but data quality is key: While larger LLMs generally perform better, the COAST framework demonstrates that high-quality, synthesized data can significantly boost the performance of smaller LLMs.
  • CoT for explanation, not necessarily repair: Chain-of-Thought reasoning is beneficial for explaining bugs (Bug Identification, Localization) but can sometimes introduce noise and hinder performance in the Code Repair task.
  • Code-specific LLMs benefit most: Code-oriented LLMs like DeepSeek-Coder show more significant improvements when fine-tuned with COAST-generated data compared to general-purpose LLMs.

Key Results

  • DEBUGEVAL reveals scale disparity: 7B-scale LLMs consistently underperform compared to larger models (e.g., 70B+) across all DEBUGEVAL tasks, highlighting their limitations in code semantics.
  • COAST significantly enhances 7B LLMs: Fine-tuning 7B-scale LLMs (like DeepSeek-Coder-6.7B-Ins and Llama3-8B-Ins) with COAST-generated data leads to substantial performance improvements. For example, NeuDebugger-DS-6.7B shows a 27.7% improvement over its base model.
  • NeuDebugger rivals GPT-3.5: The NeuDebugger models, trained with COAST data, achieve debugging performance comparable to GPT-3.5-Turbo.
  • COAST data outperforms others: Data generated by COAST is shown to be more effective than human-curated or GPT-4-generated data for fine-tuning.
  • Task-specific performance varies: LLMs generally perform better on Bug Localization and Code Recognition than on Bug Identification and Code Repair, indicating areas where further improvement is needed.

Limitations and Future Work

  • Limitations of COAST: The effectiveness of COAST is dependent on the performance and diversity of the Code Quizzer and Code Teacher models. If these foundational models are limited, the quality of synthesized data will be constrained.
  • DEBUGEVAL scope: While comprehensive, DEBUGEVAL could be further enhanced by incorporating additional tasks to provide an even more holistic evaluation of LLM debugging capabilities.
  • Future Work: The authors suggest exploring more sophisticated multi-agent interactions within COAST and expanding DEBUGEVAL to include more diverse tasks and programming languages. Further research into improving LLMs' performance on complex bug types like Logical Errors and Multiple Errors is also warranted.

Summary updated

Abstract

Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human debugging typically follows a multi-stage process, which includes Bug Localization, Bug Identification, Code Repair, and Code Recognition. However, existing code debugging benchmarks predominantly focus on the Code Repair stage, which offers only a limited perspective on evaluating the debugging capabilities of LLMs. In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process. Through evaluating on DEBUGEVAL, we observe that 7B-scale models consistently underperform compared to their larger counterparts, highlighting their limitations in comprehending code semantics. In this case, we propose the COmmunicative Agent-based data SynThesis (COAST) framework, which employs a multi-agent system to generate high-quality training data for supervised fine-tuning (SFT). Experimental results demonstrate that COAST-generated data outperform human-curated and GPT-4-generated data, enabling 7B-scale LLMs to achieve debugging performance comparable to GPT-3.5. All data and codes are available at https://github.com/NEUIR/COAST.
BibTeX
@inproceedings{yang2025coast,
  author = {Weiqing Yang and Hanbin Wang and Zhenghao Liu and Xinze Li and Yukun Yan and Shuo Wang and Yu Gu and Minghe Yu and Zhiyuan Liu and Ge Yu},
  title = {{COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.139},
}
Figure