Testing and Enhancing Multi-Agent Systems for Robust Code Generation

Zongyi Lyu, Songqiang Chen, Zhenlan Ji, Liwen Wang, Shuai Wang, Daoyuan Wu, Wenxuan Wang, Shing-Chi Cheung

Abstract

Multi-agent systems (MASs) have emerged as a promising paradigm for automated code generation, demonstrating impressive performance on established benchmarks by decomposing complex coding tasks across specialized agents with different roles. Despite their prosperous development and adoption, their robustness remains pressingly under-explored, raising critical concerns for real-world deployment. This paper presents the first comprehensive study examining the robustness of MASs for code generation through a fuzzing-based testing approach. By designing a fuzzing pipeline incorporating semantic-preserving mutation operators and a novel fitness function, we assess mainstream MASs across multiple datasets and LLMs. Our findings reveal substantial robustness flaws of various popular MASs: they fail to solve 7.9%-83.3% of problems they initially resolved successfully after applying the semantic-preserving mutations. Through comprehensive failure analysis, we identify a common yet largely overlooked cause of the robustness issue: miscommunications between planning and coding agents, where plans lack sufficient detail and coding agents misinterpret intricate logic, aligning with the challenges inherent in a multi-stage information transformation process. Accordingly, we also propose a repairing method that encompasses multi-prompt generation and introduces a new monitor agent to address this issue. Evaluation shows that our repairing method effectively enhances the robustness of MASs by solving 40.0%-88.9% of identified failures. Our work uncovers critical robustness flaws in MASs and provides effective mitigation strategies, contributing essential insights for developing more reliable MASs for code generation.

Introduction: The Robustness Challenge in MAS

Large Language Models (LLMs) have significantly advanced automated code generation. To push performance further, researchers have developed Multi-Agent Systems (MAS) that divide the coding process into specialized stages—typically planning, coding, and refinement—handled by distinct agent roles. While these systems excel on static benchmarks, their real-world robustness is often overlooked.

Multi-Agent System Architecture for Code Generation

This paper addresses a critical "black box" problem: MAS often fail when faced with semantically equivalent variations of a requirement. For instance, a simple rephrasing of a prompt can cause a system that previously succeeded to fail entirely. This fragility poses significant risks for deployment in environments where user input is naturally diverse and ambiguous.

Methodology: Fuzzing and Repairing

The researchers developed a two-phase approach to systematically evaluate and improve MAS robustness.

1. Fuzzing Framework

To identify flaws, the team built a mutation-based testing pipeline. They utilized four semantic-preserving operators—Rephrase, Insert, Expand, and Condense—to generate diverse but logically identical prompts. A novel dual-component fitness function was employed to track both the final code quality and "semantic drift" in intermediate plans, allowing the researchers to pinpoint exactly where the multi-agent communication broke down.

2. The Repairing Method

To bridge the identified gaps, the study proposes two synergistic components:

  • Multi-Prompt Generation: Executing the MAS with multiple mutated versions of the input to reduce the impact of any single "bad" interpretation.
  • Monitor Agent Insertion: A new agent role placed between the Planner and the Coder. This agent performs Plan Interpretation (clarifying core concepts and edge cases for the coder) and Code Check (a static inspection to ensure the generated code adheres to the interpreted plan).

Experimental Setup

The study evaluated three mainstream MAS—SCCG, MetaGPT, and PairCoder—using backend LLMs like GPT-3.5, GPT-4o, and DeepSeek. Testing was conducted across four diverse datasets:

  • HumanEval & MBPP: Standard Python task benchmarks.
  • CodeContest: Challenging competitive programming problems requiring complex logic.
  • CoderEval: Real-world code sourced from GitHub repositories.

Key Results and Insights

Fragility Exposed

The results were striking: applying semantic-preserving mutations caused MAS performance (Pass@10) to drop by 7.9% to 83.3%. The most severe degradations occurred on complex datasets like CodeContest, where intricate logic is easily lost in translation between agents.

Robustness Evaluation Results across MAS and Datasets

The Planner-Coder Gap

A deep dive into the failures revealed that 75.3% of robustness issues stem from the "gap" between agents rather than faulty plans. Planners often produce high-level, abstract plans that are logically sound but "underspecified." This leads the coding agent to misinterpret the intent, resulting in five common Error Patterns (EPs), such as misunderstanding core concepts or omitting specific logic paths.

Effective Recovery

The proposed repairing method successfully recovered 40.0%–88.9% of the previously failed cases. Furthermore, the monitor agent adds minimal overhead, typically requiring only 2.7s–3.7s per attempt, which is negligible compared to the total execution time of a standard MAS.

Conclusion and Future Work

This research uncovers a fundamental vulnerability in multi-agent code generation: cumulative information loss across agent boundaries. By introducing a "monitor agent" as a quality gate, the researchers demonstrated that these systems can be made significantly more robust with minimal overhead.

Future work will explore incorporating dynamic execution feedback into the monitor agent's loop and investigating similar communication gaps between other agent pairs, such as the coder and the tester, to further harden these systems for production use.

Summary updated

Abstract

Multi-agent systems (MASs) have emerged as a promising paradigm for automated code generation, demonstrating impressive performance on established benchmarks by decomposing complex coding tasks across specialized agents with different roles. Despite their prosperous development and adoption, their robustness remains pressingly under-explored, raising critical concerns for real-world deployment. This paper presents the first comprehensive study examining the robustness of MASs for code generation through a fuzzing-based testing approach. By designing a fuzzing pipeline incorporating semantic-preserving mutation operators and a novel fitness function, we assess mainstream MASs across multiple datasets and LLMs. Our findings reveal substantial robustness flaws of various popular MASs: they fail to solve 7.9%-83.3% of problems they initially resolved successfully after applying the semantic-preserving mutations. Through comprehensive failure analysis, we identify a common yet largely overlooked cause of the robustness issue: miscommunications between planning and coding agents, where plans lack sufficient detail and coding agents misinterpret intricate logic, aligning with the challenges inherent in a multi-stage information transformation process. Accordingly, we also propose a repairing method that encompasses multi-prompt generation and introduces a new monitor agent to address this issue. Evaluation shows that our repairing method effectively enhances the robustness of MASs by solving 40.0%-88.9% of identified failures. Our work uncovers critical robustness flaws in MASs and provides effective mitigation strategies, contributing essential insights for developing more reliable MASs for code generation.
BibTeX
@misc{lyu2025testing,
  author = {Zongyi Lyu and Songqiang Chen and Zhenlan Ji and Liwen Wang and Shuai Wang and Daoyuan Wu and Wenxuan Wang and Shing-Chi Cheung},
  title = {{Testing and Enhancing Multi-Agent Systems for Robust Code Generation}},
  year = {2025},
  eprint = {2510.10460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10460},
}
Figure