On the Resilience of Multi-Agent Systems with Malicious Agents

Jen-tse Huang, Jiaxu Zhou, Tailin Jin, Xuhui Zhou, Zixi Chen, Wenxuan Wang, Youliang Yuan, Maarten Sap, Michael R. Lyu

Abstract

Multi-agent systems (MAS) leveraging Large Language Models (LLMs) have demonstrated significant potential in solving complex tasks. However, their collaborative nature introduces vulnerabilities to "faulty" agents—those that are either unintentionally error-prone (clumsy) or intentionally sabotaging (malicious). This paper investigates the resilience of various MAS organizational structures (Linear, Flat, and Hierarchical) against such agents. Our findings indicate that hierarchical structures are the most robust. Furthermore, we introduce two defense mechanisms, the Challenger and the Inspector, which can recover up to 96.4% of performance lost to errors.

Introduction

Large Language Model (LLM) multi-agent systems have become a standard architecture for complex problem-solving in domains like software engineering and mathematics. By decomposing problems into specialized sub-tasks, these systems often surpass the capabilities of single-model approaches.

Despite their success, a critical security and reliability gap exists: the impact of a single "clumsy" or "malicious" agent. In decentralized environments, a single bad actor can trigger a "failure cascade," where erroneous code or logic is propagated and amplified by the collective. This study aims to identify which system structures best withstand these failures and proposes "immune systems" to mitigate their impact.

Methods

To evaluate MAS resilience, the researchers developed two novel methods for introducing controlled errors:

  1. AUTOTRANSFORM: This method modifies an agent's system prompt (its "profile") to convert it into a "vicious" agent. The agent is instructed to subtly introduce errors that appear correct at first glance but lead to incorrect final results.
  2. AUTOINJECT: This method provides granular control by intercepting messages and injecting specific syntactic or semantic errors at a defined rate.
Overview of the AutoTransform and AutoInject frameworks for introducing errors into multi-agent systems.

To counter these threats, two defense strategies were proposed:

  • The Challenger: A profile modification that empowers agents to perform sanity checks and challenge incoming outputs from peers.
  • The Inspector: A specialized "police" agent that monitors chat history, identifies errors, and corrects faulty responses before they propagate.

Datasets and Evaluation

The study benchmarked six multi-agent frameworks (e.g., MetaGPT, Camel, MAD) across four diverse tasks:

  • HumanEval: Python code generation.
  • CIAR: Counter-intuitive arithmetic problem solving.
  • CommonMT: Text translation with ambiguous lexical contexts.
  • FairEval: Quality evaluation of LLM responses.

Key Insights and Ideas

The organizational structure of an MAS plays a pivotal role in its resilience.

Comparison of Linear, Flat, and Hierarchical multi-agent structures and their resilience to faulty agents.
  • Hierarchy as a Defense: Hierarchical structures, featuring a "leader" or "judge" overseeing "workers," proved significantly more resilient than Linear or Flat structures. The centralized oversight allows for cross-verification of sub-tasks, making it harder for single errors to persist.
  • Semantic vs. Syntactic Errors: While MAS are adept at catching syntactic errors (e.g., broken code) through feedback loops, semantic errors (logic bugs) are far more insidious. These "stealthy" errors often go undetected and mislead the entire system.
  • The "Stagnation" Paradox: In certain debate-heavy systems, injected errors occasionally improved performance by acting as "stochastic noise," helping agents escape repetitive loops or local minima.

Key Results

  • Resilience Ranking: Hierarchical structures experienced the smallest performance decline (5.5%), compared to Flat (10.5%) and Linear structures (23.7%).
  • Task Vulnerability: Code generation was identified as the most sensitive task, with accuracy dropping by 22.6% in the presence of malicious agents.
  • Defense Efficacy: The combination of the Challenger and Inspector recovered 96.4% of lost performance in highly vulnerable systems like the Linear "Self-collab" framework.
  • Model Scaling: While GPT-4o improved overall resilience over GPT-3.5, the relative superiority of hierarchical structures remained constant.

Limitations and Future Work

The study notes several areas for future exploration:

  • Model Diversity: Research was limited to GPT-3.5 and GPT-4o; the resilience of open-source models is yet to be determined.
  • Multimodal MAS: Future work should investigate vulnerabilities in agents handling non-textual data like images or web navigation.
  • Human-in-the-Loop: Automated scoring may miss subtle manipulative behaviors that could affect human users.
  • Prompt Bias: Results are inherently linked to the specific prompt designs used in the studied frameworks.

Summary updated

BibTeX
@inproceedings{huang2025resilience,
  author = {Jen-tse Huang and Jiaxu Zhou and Tailin Jin and Xuhui Zhou and Zixi Chen and Wenxuan Wang and Youliang Yuan and Maarten Sap and Michael R. Lyu},
  title = {{On the Resilience of Multi-Agent Systems with Malicious Agents}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=bkiM54QftZ},
}
Figure