Deep reinforcement learning has achieved remarkable success in single-agent tasks, and its extension to multi-agent reinforcement learning (MARL) promises to revolutionize cooperative swarms, autonomous driving, and network defense. However, as noted early on by Donghwan Lee et al. in Optimization for Reinforcement Learning: From a single agent to cooperative agents, the resilience of these distributed systems against malicious actors remains a critical blind spot.
Securing MARL is not simply a matter of scaling up single-agent defenses. In a multi-agent system, agents do not just interact with a static environment; they interact with each other. This interdependence creates emergent attack surfaces where small, localized disruptions can cascade into systemic failures.
The Emergent Multi-Agent Attack Surface
In single-agent reinforcement learning, an adversary typically perturbs the agent's sensory inputs or rewards to induce a mistake. In MARL, the threat vector expands exponentially because agents are deeply coupled.
When agents learn to cooperate, they form implicit or explicit dependencies on their teammates' actions and communication signals. In a comprehensive survey, Maxwell Standen et al. highlight in Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement Learning that standard Markov Decision Processes (MDPs) fail to capture these nuances. Because cooperative policies rely on coordinated decision-making, an attacker does not need to compromise the entire system. Altering a shared observation or compromising a single agent can break the team's coordination, causing the entire system to collapse.
Exploiting Coordination via Model-Based and Traitor Attacks
To understand why cooperative MARL is uniquely fragile, consider how adversaries can leverage the environment's underlying structure. Rather than relying on computationally expensive model-free perturbations, Nhan H Pham et al. introduced a model-based attack framework in c-MBA: Adversarial Attack for Cooperative MARL Using Learned Dynamics Model. By learning a predictive model of the environment's dynamics, an attacker can precisely calculate state perturbations that maximize team degradation with minimal input changes.
In subsequent work, Attacking c-MARL More Effectively: A Data Driven Approach, Nhan H. Pham et al. demonstrated that this model-based approach allows attackers to strategically select specific "victim agents" to target. By exploiting the complex inter-agent dependencies inherent in cooperative MARL, the attack achieves high potency while remaining highly constrained.
[Cooperative Team] ---> (Inter-agent Dependencies) ---> [Shared Goal]
^
| (Targeted Perturbation)
[Model-Based Attack]
Alternatively, an adversary can attack from within. Instead of manipulating external observations, the framework proposed by Zhen Chen 0025 et al. in CuDA2: An Approach for Incorporating Traitor Agents Into Cooperative Multiagent Systems trains "traitor agents." These traitors operate within the cooperative team but use curiosity-driven exploration to discover physical or behavioral bottlenecks—such as strategic collisions or blocking positions—that subtly derail the team's objectives without requiring direct access to the victim agents' internal policies.
The Illusion of Partial Observability
A common intuition in system design is that partial observability acts as a natural defense mechanism: if an attacker cannot see the entire environment, they cannot effectively disrupt it. However, recent research explicitly dismantles this assumption. David Hubczenko et al. demonstrated in Adversarial Reinforcement Learning under Partial Observability in Autonomous Computer Network Defence that even with limited visibility, adversaries can severely compromise agent training.
To formalize this threat, Oubo Ma et al. developed a black-box attack framework called SUB-PLAY in Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems. SUB-PLAY works by decomposing a complex, global game into localized "subgames." By sharing training data across these subgames, the adversarial policies learn to exploit the victim swarm's coordinated patterns using only noisy, incomplete, or local observations. This proves that limited sensing capabilities do not safeguard a system; rather, they merely force the adversary to exploit local coordination patterns.
Poisoning the Learning Process and Faking Equilibria
Attacks are not limited to deployment time. Evaluating security solely at deployment may overestimate a system's robustness because it ignores how agents adapt during training. To address this, Siyang Wu et al. introduced training-time attacks in Model and Method: Training-Time Attack for Cooperative Multi-Agent Reinforcement Learning. By keeping the adversarial perturbations small enough to blend into normal exploration noise, the attacker stealthily misguides the entire policy optimization process.
Furthermore, in game-theoretic settings, attackers can manipulate the very concept of optimality. Young Wu et al. showed in Data Poisoning to Fake a Nash Equilibria for Markov Games that offline MARL is highly vulnerable to data poisoning. The authors provided a mathematical framework where an attacker can modify rewards or transitions to trick agents into converging to a completely fabricated, malicious Nash Equilibrium.
[True Data] -----\
+---> [Poisoned Training] ---> [Fictitious Nash Equilibrium]
[Malicious Data] -/
Similarly, Guanlin Liu et al. proved in Efficient Adversarial Attacks on Online Multi-agent Reinforcement Learning that a mixed attack strategy—simultaneously poisoning rewards and actions—can force online agents into a target adversarial equilibrium with sub-linear costs, even in a black-box setting where the attacker knows nothing about the environment or the underlying algorithms.
Rethinking Multi-Agent Defense
Traditional single-agent defense mechanisms, such as explicit adversarial training or exhaustive scenario enumeration, fail in MARL. As noted by Simin Li et al. in Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization, these approaches are computationally prohibitive due to the exponential explosion of multi-agent state-action spaces, and they often result in overly pessimistic policies that generalize poorly.
To build genuine resilience, researchers are moving toward intrinsic regularization and adaptive defense:
- Mutual Information Regularization: Instead of training against explicit adversaries, the MIR3 framework by Simin Li et al. frames robustness as an inference problem. By regularizing mutual information, agents learn to remain resilient to unexpected or adversarial actions from teammates without ever undergoing explicit adversarial training.
- Inversion Defense: On the defensive side of network security, the mechanism introduced by David Hubczenko et al. in Adversarial Reinforcement Learning under Partial Observability in Autonomous Computer Network Defence mitigates attacks by learning to reverse the adversary's state manipulations dynamically, even without knowing the attacker's exact strategy.
Developing these inherently robust, equilibrium-stable algorithms remains the primary frontier in securing multi-agent reinforcement learning against sophisticated, coordinated threats.
Key Foundational Work
c-MBA: Adversarial Attack for Cooperative MARL Using Learned Dynamics Model (2022) — contributed 2 concept elements to this direction.
Model and Method: Training-Time Attack for Cooperative Multi-Agent Reinforcement Learning (2022) — contributed 2 concept elements to this direction.
Optimization for Reinforcement Learning: From a single agent to cooperative agents (2020) — contributed 1 concept elements to this direction.