TL;DR
This paper exposes a critical vulnerability in Large Language Model (LLM)-based multi-agent systems: malicious agents can iteratively optimize their responses to manipulate the system's final output. To counter this, the authors introduce SecMAS, a secure aggregation rule that uses semantic similarity to identify and select the most "trustworthy" response from a group, effectively neutralizing even sophisticated, adaptive attacks.
What Problem Does This Paper Address?
Multi-agent systems (MAS) are increasingly used to solve complex problems by breaking them into subtasks. A "manager agent" decomposes a task, delegates subtasks to groups of specialized agents, and then uses an aggregation rule to combine their responses into a final solution.
The security gap lies in that aggregation step. If an attacker controls even a small minority of agents within a group, they can provide "malicious responses" designed to steer the manager agent toward an incorrect or specifically chosen wrong answer. Existing attacks were often simple heuristics, leading to a false sense of security. This paper aims to show that with systematic optimization, these systems are highly vulnerable, and it proposes a "secure-by-design" aggregation method to fix it.
Methods
The Attack: ReMA (Response Manipulation Attacks)
The authors developed ReMA, an optimization-based attack. Instead of just sending a random wrong answer, ReMA uses a "surrogate" multi-agent system to iteratively refine malicious responses.
The process involves three LLM-based components:
- Response Generator: Crafts candidate malicious responses.
- Surrogate Manager Agent: Simulates the target system's aggregation process.
- Attack Critic: Evaluates the resulting solution and provides "reflective feedback" (e.g., "This response was too obvious; try embedding the error in plausible reasoning").
The Defense: SecMAS (Secure Multi-Agent System)
SecMAS is built on a simple but powerful intuition: benign agents are usually semantically consistent with each other, while malicious agents are outliers.
For every subtask, SecMAS:
- Converts all agent responses into high-dimensional embeddings.
- Calculates pairwise cosine similarities between all responses.
- Assigns a response score to each agent based on its similarity to a "sufficiently large subset" of other agents.
- Selects the response with the highest score as the "truth" for that subtask.
Datasets
Because existing benchmarks like TAMAS are often open-ended, the authors created a new evaluation dataset:
- 500 Task-Solution Pairs: Spanning five scenarios: education, finance, healthcare, legal, and news.
- Generation & Verification: Tasks and ground-truth solutions were generated using GPT-5.6-Sol and verified for correctness by Claude Opus 5.
- Diverse Agents: The evaluation used a mix of six different backbone LLMs (including Llama, Qwen, and Gemma) and two agent frameworks (AutoGen and CrewAI) to ensure results weren't model-specific.
Key Insights and Ideas
- Optimization Trumps Heuristics: Traditional "red-teaming" often relies on manual prompt injection. ReMA shows that an automated, iterative feedback loop can find subtle vulnerabilities that humans might miss, reaching a 100% attack success rate against standard aggregation rules.
- Semantic Consensus as a Shield: By moving from simple text summarization to embedding-based similarity, SecMAS treats security as a "consensus" problem. It effectively applies principles from Byzantine fault tolerance (common in distributed systems) to the world of natural language.
- Theoretical Guarantees: The authors don't just show empirical success; they provide mathematical proofs (lower and upper bounds) for SecMAS's performance, showing that as long as benign agents are reasonably competent and in the majority, the system remains secure.
Key Results
- Vulnerability of Baselines: Standard aggregation rules (like simple LLM summarization or confidence-based voting) were highly vulnerable. ReMA achieved an Attack Success Rate (ASR) of 1.00 in several white-box settings.
- SecMAS Effectiveness: SecMAS reduced the ASR to nearly 0.00 across all tested scenarios, even against "adaptive" versions of ReMA where the attacker knew exactly how SecMAS worked.
- Utility Preservation: Crucially, SecMAS maintained "Performance Under No Attack" (PNA) comparable to baseline rules. It provides security without sacrificing the system's ability to solve complex tasks correctly.
Limitations and Future Work
- The "Benign Majority" Assumption: SecMAS relies on the assumption that the majority of agents in a group are benign and capable of producing semantically similar, correct answers. If an attacker compromises more than half the agents, the consensus mechanism could be subverted.
- Quality Variance: If benign agents produce highly diverse (but correct) answers—common in creative writing tasks—the semantic similarity approach might struggle to find a clear consensus.
- Future Directions: The authors suggest exploring even stronger adaptive attacks and investigating how to maintain security when the quality of benign agents varies significantly.
Summary updated