Securing Multi-Agent Systems by Design

Xilong Wang, Xiaoxing Chen, Neil GongarXiv 2026 · arXiv

TL;DR

This paper exposes a critical vulnerability in Large Language Model (LLM)-based multi-agent systems: malicious agents can iteratively optimize their responses to manipulate the system's final output. To counter this, the authors introduce SecMAS, a secure aggregation rule that uses semantic similarity to identify and select the most "trustworthy" response from a group, effectively neutralizing even sophisticated, adaptive attacks.

What Problem Does This Paper Address?

Multi-agent systems (MAS) are increasingly used to solve complex problems by breaking them into subtasks. A "manager agent" decomposes a task, delegates subtasks to groups of specialized agents, and then uses an aggregation rule to combine their responses into a final solution.

Figure 1: Illustration of the three steps performed in each iteration of multi-agent systems.

The security gap lies in that aggregation step. If an attacker controls even a small minority of agents within a group, they can provide "malicious responses" designed to steer the manager agent toward an incorrect or specifically chosen wrong answer. Existing attacks were often simple heuristics, leading to a false sense of security. This paper aims to show that with systematic optimization, these systems are highly vulnerable, and it proposes a "secure-by-design" aggregation method to fix it.

Methods

The Attack: ReMA (Response Manipulation Attacks)

The authors developed ReMA, an optimization-based attack. Instead of just sending a random wrong answer, ReMA uses a "surrogate" multi-agent system to iteratively refine malicious responses.

Figure 2: Illustration of our iterative method for optimizing malicious responses in each malicious iteration.

The process involves three LLM-based components:

  1. Response Generator: Crafts candidate malicious responses.
  2. Surrogate Manager Agent: Simulates the target system's aggregation process.
  3. Attack Critic: Evaluates the resulting solution and provides "reflective feedback" (e.g., "This response was too obvious; try embedding the error in plausible reasoning").

The Defense: SecMAS (Secure Multi-Agent System)

SecMAS is built on a simple but powerful intuition: benign agents are usually semantically consistent with each other, while malicious agents are outliers.

For every subtask, SecMAS:

  1. Converts all agent responses into high-dimensional embeddings.
  2. Calculates pairwise cosine similarities between all responses.
  3. Assigns a response score to each agent based on its similarity to a "sufficiently large subset" of other agents.
  4. Selects the response with the highest score as the "truth" for that subtask.

Datasets

Because existing benchmarks like TAMAS are often open-ended, the authors created a new evaluation dataset:

  • 500 Task-Solution Pairs: Spanning five scenarios: education, finance, healthcare, legal, and news.
  • Generation & Verification: Tasks and ground-truth solutions were generated using GPT-5.6-Sol and verified for correctness by Claude Opus 5.
  • Diverse Agents: The evaluation used a mix of six different backbone LLMs (including Llama, Qwen, and Gemma) and two agent frameworks (AutoGen and CrewAI) to ensure results weren't model-specific.

Key Insights and Ideas

  • Optimization Trumps Heuristics: Traditional "red-teaming" often relies on manual prompt injection. ReMA shows that an automated, iterative feedback loop can find subtle vulnerabilities that humans might miss, reaching a 100% attack success rate against standard aggregation rules.
  • Semantic Consensus as a Shield: By moving from simple text summarization to embedding-based similarity, SecMAS treats security as a "consensus" problem. It effectively applies principles from Byzantine fault tolerance (common in distributed systems) to the world of natural language.
  • Theoretical Guarantees: The authors don't just show empirical success; they provide mathematical proofs (lower and upper bounds) for SecMAS's performance, showing that as long as benign agents are reasonably competent and in the majority, the system remains secure.

Key Results

  • Vulnerability of Baselines: Standard aggregation rules (like simple LLM summarization or confidence-based voting) were highly vulnerable. ReMA achieved an Attack Success Rate (ASR) of 1.00 in several white-box settings.
  • SecMAS Effectiveness: SecMAS reduced the ASR to nearly 0.00 across all tested scenarios, even against "adaptive" versions of ReMA where the attacker knew exactly how SecMAS worked.
  • Utility Preservation: Crucially, SecMAS maintained "Performance Under No Attack" (PNA) comparable to baseline rules. It provides security without sacrificing the system's ability to solve complex tasks correctly.

Limitations and Future Work

  • The "Benign Majority" Assumption: SecMAS relies on the assumption that the majority of agents in a group are benign and capable of producing semantically similar, correct answers. If an attacker compromises more than half the agents, the consensus mechanism could be subverted.
  • Quality Variance: If benign agents produce highly diverse (but correct) answers—common in creative writing tasks—the semantic similarity approach might struggle to find a clear consensus.
  • Future Directions: The authors suggest exploring even stronger adaptive attacks and investigating how to maintain security when the quality of benign agents varies significantly.

Summary updated

Abstract

Multi-agent systems solve complex tasks by decomposing them into subtasks, delegating the subtasks to agents, and aggregating the agents' responses using an aggregation rule. Malicious agents can manipulate the final solution by providing malicious responses to their assigned subtasks. In this paper, we first propose response manipulation attacks (ReMA), which iteratively optimize malicious responses to maximize attack effectiveness. Specifically, we formulate untargeted and targeted attacks as optimization problems and develop an approach to approximately solve these problems by iteratively refining malicious responses based on feedback from a surrogate multi-agent system. To defend against such attacks, we further propose SecMAS, a secure aggregation rule that leverages the semantic similarities among agent responses to select the response that is most consistent with a sufficiently large subset of the other responses. We theoretically establish security guarantees for SecMAS against both untargeted and targeted attacks. Extensive evaluations demonstrate that existing aggregation rules are vulnerable to ReMA, whereas SecMAS effectively defends against all evaluated attacks, including adaptive variants of ReMA, while maintaining utility comparable to that of existing aggregation rules.
BibTeX
@misc{wang2026securing,
  author = {Xilong Wang and Xiaoxing Chen and Neil Gong},
  title = {{Securing Multi-Agent Systems by Design}},
  year = {2026},
  eprint = {2611.17883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17883},
}
Figure