Robust Communicative Multi-Agent Reinforcement Learning with Active Defense

Lebin Yu, Yunbo Qiu, Quanming Yao, Yuan Shen, Xudong Zhang, Jian Wang

Introduction

Multi-Agent Reinforcement Learning (MARL) relies heavily on communication to coordinate complex tasks, from warehouse robotics to drone swarm navigation. However, these communication channels are vulnerable "soft targets." They are susceptible to both random environmental noise and deliberate adversarial attacks, where malicious agents send deceptive data to induce system failure.

Traditional approaches to this problem typically employ passive defense strategies. These methods attempt to train policies that are inherently robust to a certain level of data corruption. However, they often suffer from a "garbage in, garbage out" limitation: if the input is sufficiently corrupted, the policy inevitably fails. Furthermore, passive robustness often comes at the cost of performance in clean, non-adversarial environments.

The ADMAC Framework

To address these vulnerabilities, researchers have introduced the Active Defense Multi-Agent Communication (ADMAC) framework. Unlike passive systems that treat all incoming messages as equally valid, ADMAC enables agents to "fact-check" communications against their own local observations.

The ADMAC framework shifts from passive defense (treating all messages as valid) to active defense, where a reliability judge filters incoming messages before they reach the policy network.

ADMAC is built upon two core technical pillars:

  1. Reliability Estimator: A classifier that evaluates each incoming message. By comparing a message (e.g., "the target is at coordinates X, Y") against the agent's local observations and internal memory (hidden states), it assigns a reliability weight between 0 and 1. This allows the agent to identify and down-weight messages that contradict its own "local truth."
  2. Decomposable Message Aggregation Policy Net: Standard neural architectures often aggregate input data in a way that makes it difficult to isolate individual corrupted signals. ADMAC utilizes a decomposable structure where the final decision is a summation of a "base preference" (derived from local data) and several "message-specific preferences." The reliability weight directly scales the impact of each specific message on the final action distribution.

Training and Evaluation

The ADMAC training pipeline involves three stages: first, training the policy in a clean environment to establish a gold standard for cooperation; second, generating a dataset by subjecting these agents to various attacks and labeling messages as "reliable" or "unreliable" based on their utility; and third, training the Reliability Estimator using supervised learning on this labeled data.

Performance was evaluated across three communication-critical environments: Food Collector, Predator Prey, and Treasure Hunt. These environments were subjected to diverse attack types, including Gaussian noise and sophisticated adversarial methods like Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD).

Key Results and Insights

ADMAC demonstrates superior robustness compared to existing baselines like TARMAC (no defense) and AME (consensus-based defense). A critical finding is ADMAC's resilience to high-intensity attacks.

Performance comparison in the Food Collector environment under varying attack probabilities. ADMAC (blue) maintains stable, low completion times (timesteps) even as attack intensity increases, whereas other methods degrade significantly.

As shown in the experimental results, even when 30% of messages are maliciously corrupted ($p=0.3$), ADMAC's performance remains stable. In contrast, other methods see their completion times skyrocket. Furthermore, the reliability estimator achieved high precision and recall (e.g., 0.90/0.86 in Food Collector), proving that agents can effectively learn to identify deceptive communications without becoming overly "paranoid" and ignoring useful information.

Conclusion

ADMAC represents a significant step toward secure and robust multi-agent systems. By shifting the defensive burden from the policy's general "toughness" to an active reliability judgment, the framework maintains high performance in both peaceful and adversarial settings. Future work will focus on extending this framework to continuous control spaces and improving scalability for larger agent swarms where the computational overhead of individual message judgment may become a factor.

Summary updated

Abstract

Communication in multi-agent reinforcement learning (MARL) has been proven to effectively promote cooperation among agents recently. Since communication in real-world scenarios is vulnerable to noises and adversarial attacks, it is crucial to develop robust communicative MARL technique. However, existing research in this domain has predominantly focused on passive defense strategies, where agents receive all messages equally, making it hard to balance performance and robustness. We propose an active defense strategy, where agents automatically reduce the impact of potentially harmful messages on the final decision. There are two challenges to implement this strategy, that are defining unreliable messages and adjusting the unreliable messages' impact on the final decision properly. To address them, we design an Active Defense Multi-Agent Communication framework (ADMAC), which estimates the reliability of received messages and adjusts their impact on the final decision accordingly with the help of a decomposable decision structure. The superiority of ADMAC over existing methods is validated by experiments in three communication-critical tasks under four types of attacks.
BibTeX
@inproceedings{yu2024robust,
  author = {Lebin Yu and Yunbo Qiu and Quanming Yao and Yuan Shen and Xudong Zhang and Jian Wang},
  title = {{Robust Communicative Multi-Agent Reinforcement Learning with Active Defense}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29708},
}
Figure