COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis

Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, Ge Yu

TL;DR

This paper introduces DEBUGEVAL, a new multi-stage benchmark that mimics the human debugging process (localization, identification, repair, and recognition) to better evaluate LLM coding capabilities. To address the poor performance of 7B-scale models on this benchmark, the authors develop COAST, a multi-agent data synthesis framework that generates high-quality training data, elevating smaller models to debugging performance on par with GPT-3.5.

What Problem Does This Paper Address?

When humans debug code, they rarely just rewrite a broken function from scratch in one shot. Instead, they follow a systematic, multi-stage workflow: locating the buggy line, analyzing the root cause, applying a fix, and verifying the corrected code against the original.

Figure 1: Illustration of the Human Code Debugging Process. Existing studies (Olausson et al., 2023; Chen et al., 2023) typically focus on directly repairing code generated by Large Language Models (LLMs). In con- trast to this approach, humans often engage in a multi- stage process to resolve buggy codes.

However, existing code debugging benchmarks predominantly focus only on the Code Repair stage. This creates a significant evaluation gap. Large Language Models (LLMs) might be able to patch a simple error by pattern matching, but do they actually understand code semantics and error types? The authors show that smaller, 7B-scale models consistently struggle with holistic debugging, frequently failing to identify or locate errors even when they can occasionally fix them.

Methods

To bridge this gap, the authors propose the COmmunicative Agent-based data SynThesis (COAST) framework, which uses a three-agent cooperative system to generate high-quality Supervised Fine-Tuning (SFT) data:

  1. Code Quizzer: Utilizing a powerful model (DeepSeek-Coder-V2), this agent generates diverse code debugging problems based on seed examples.
  2. Code Learner: This agent uses the same backbone as the target model being trained. It attempts to solve the Quizzer's problems and acts as a filter. If the Learner answers a question correctly, the problem is discarded. If it answers incorrectly, the problem is flagged as having high "educational value" for fine-tuning.
  3. Code Teacher: A strong LLM agent that reviews the failed problems and generates detailed Chain-of-Thought (CoT) explanations and correct answers to guide the Learner.
Figure 3: Illustration of COmmunicative Agent Based Data SynThesis (COAST) Framework.

Crucially, the authors discovered that while CoT reasoning helps significantly with localization and identification tasks, it actually hurts Code Repair performance by introducing structural noise into the code. Therefore, their final NeuDebugger models are trained using a hybrid strategy: CoT data for localization, identification, and recognition, but direct code answers for the repair task.

Datasets

The authors introduced and utilized the following datasets:

  • DEBUGEVAL (New Benchmark): Contains 5,712 test instances spanning three programming languages (Python, C++, Java) and 18 distinct error types. The bugs are sourced from both real-world human trials (AtCoder) and GPT-4 generations to simulate realistic software development challenges.
  • COAST Synthesized Data: A training dataset consisting of 4,681 Bug Localization, 4,474 Bug Identification, 11,317 Code Repair, and 4,420 Code Recognition instances.
  • Vanilla SFT Baselines: For comparison, models were also trained on existing datasets like UltraInteract, InstructCoder, and RepairLlama.

Key Insights and Ideas

  • Failure-Driven Learning: Filtering synthetic data based on what the student model fails to understand creates a highly efficient curriculum, outperforming standard human-curated or un-filtered GPT-4 datasets.
  • The CoT Trade-Off in Coding: Chain-of-Thought is a double-edged sword. It provides the logical scaffolding needed to classify and locate bugs, but when generating raw code fixes, the extra text can disrupt the model's adherence to strict code syntax.
  • Overcoming the "Logical Error" Bias: Untrained 7B models exhibit a strong bias toward classifying almost every bug as a "Logical Error" due to a lack of nuanced understanding. COAST training successfully flattens this distribution, giving models the ability to distinguish syntax and reference errors accurately.
Figure 4: Response Distributions of DSCoder-6.7B-Ins and NeuDebugger-DS-6.7B in BUG Identification Task.

Key Results

  • NeuDebugger-DS-6.7B (fine-tuned using COAST) achieved an average debugging accuracy improvement of 27.7% over its baseline model, achieving performance comparable to GPT-3.5-Turbo across the DEBUGEVAL benchmark.
  • NeuDebugger-Llama3-8B saw a 4.1% overall improvement, demonstrating that while COAST works universally, code-specialized base models (like DeepSeek) benefit far more from debugging SFT data than general-purpose models.

Limitations and Future Work

While DEBUGEVAL expands evaluation to four distinct tasks, the authors note that incorporating even more real-world debugging scenarios (such as multi-file debugging or runtime log analysis) would make the benchmark more comprehensive. Additionally, the COAST framework's efficacy is fundamentally bounded by the capabilities of its "Teacher" and "Quizzer" models; if the underlying foundation model lacks advanced reasoning, the quality of the synthesized explanations degrades. Future work will explore scaling these agent interactions to even more complex software architectures.

Summary updated

Abstract

Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human debugging typically follows a multi-stage process, which includes Bug Localization, Bug Identification, Code Repair, and Code Recognition. However, existing code debugging benchmarks predominantly focus on the Code Repair stage, which offers only a limited perspective on evaluating the debugging capabilities of LLMs. In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process. Through evaluating on DEBUGEVAL, we observe that 7B-scale models consistently underperform compared to their larger counterparts, highlighting their limitations in comprehending code semantics. In this case, we propose the COmmunicative Agent-based data SynThesis (COAST) framework, which employs a multi-agent system to generate high-quality training data for supervised fine-tuning (SFT). Experimental results demonstrate that COAST-generated data outperform human-curated and GPT-4-generated data, enabling 7B-scale LLMs to achieve debugging performance comparable to GPT-3.5. All data and codes are available at https://github.com/NEUIR/COAST.
BibTeX
@inproceedings{yang2025coast,
  author = {Weiqing Yang and Hanbin Wang and Zhenghao Liu and Xinze Li and Yukun Yan and Shuo Wang and Yu Gu and Minghe Yu and Zhiyuan Liu and Ge Yu},
  title = {{COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.139},
}
Figure