Robust Communicative Multi-Agent Reinforcement Learning with Active Defense

Lebin Yu, Yunbo Qiu, Quanming Yao, Yuan Shen, Xudong Zhang, Jian Wang

Abstract

Communication in multi-agent reinforcement learning (MARL) has been proven to effectively promote cooperation among agents recently. Since communication in real-world scenarios is vulnerable to noises and adversarial attacks, it is crucial to develop robust communicative MARL technique. However, existing research in this domain has predominantly focused on passive defense strategies, where agents receive all messages equally, making it hard to balance performance and robustness. We propose an active defense strategy, where agents automatically reduce the impact of potentially harmful messages on the final decision. There are two challenges to implement this strategy, that are defining unreliable messages and adjusting the unreliable messages' impact on the final decision properly. To address them, we design an Active Defense Multi-Agent Communication framework (ADMAC), which estimates the reliability of received messages and adjusts their impact on the final decision accordingly with the help of a decomposable decision structure. The superiority of ADMAC over existing methods is validated by experiments in three communication-critical tasks under four types of attacks.

Introduction

In cooperative multi-agent systems, such as autonomous vehicle fleets or drone swarms, communication is essential for synchronization and efficiency. However, these communication channels are often exposed to environmental noise or deliberate adversarial attacks. Traditional approaches to securing these systems have relied on passive defense, where agents are trained to be generally robust while treating all incoming messages with equal weight. This often leads to a "robustness tax," where performance on clean data is sacrificed for stability under attack.

Conceptual comparison of passive and active defense strategies.

This paper introduces ADMAC (Active Defense Multi-Agent Communication), a framework that shifts the paradigm toward active defense. Instead of blindly accepting all data, ADMAC enables agents to use their private local observations as a "ground truth" to evaluate the reliability of incoming messages, dynamically down-weighting those that appear malicious or contradictory.

The ADMAC Framework

Implementing an active defense strategy requires solving two primary problems: identifying which messages are unreliable and adjusting their influence on the final decision without breaking the underlying policy. ADMAC addresses these through two core components:

1. Reliability Estimator ($f_R$)

Each agent is equipped with a reliability estimator that acts as a decentralized fact-checker. By comparing a received message against the agent's own local observation and historical state, the estimator outputs a reliability score between 0 and 1. If a message suggests an action that contradicts what the agent perceives locally, its reliability score is reduced.

2. Decomposable Message Aggregation

Standard MARL architectures often process messages through complex, non-linear layers, making it difficult to isolate the impact of a single bad message. ADMAC utilizes a decomposable structure where the final action preference is a weighted sum of a "base action preference" (derived from local data) and multiple "message action preferences."

Detailed architecture of the ADMAC framework, highlighting the reliability estimation and decomposable message aggregation.

As shown in the architecture, the final decision vector $v_i^t$ is calculated as: $$v_i^t = \text{Base Preference} + \sum (\text{Reliability}_j \times \text{Message Preference}_j)$$ This allows the agent to effectively "mute" a specific peer's input if it is deemed unreliable, providing a surgical defense mechanism.

Experimental Evaluation

The framework was tested across three diverse environments: Food Collector, Predator Prey, and Treasure Hunt. These tasks were subjected to four distinct attack vectors, ranging from simple Gaussian noise to sophisticated gradient-based adversarial attacks like PGD (Projected Gradient Descent).

Key Findings

  • Superior Robustness: ADMAC consistently outperformed baselines like TARMAC and standard Adversarial Training (AT). In the Treasure Hunt task under PGD attacks, ADMAC completed the objective in roughly 25 timesteps, whereas traditional methods often failed to finish within the limit.
  • Minimal Robustness Tax: Unlike passive methods, ADMAC maintains near-optimal performance in clean environments. When no attack is present, the reliability estimator correctly assigns high weights to all messages, allowing the system to function as a standard cooperative network.
  • Interpretability: The decomposable structure provides a level of transparency. Researchers can observe exactly which messages an agent chooses to ignore, offering insights into how the adversarial defense is operating in real-time.

Conclusion

ADMAC represents a significant step forward in securing multi-agent systems. By empowering agents to actively judge the reliability of their peers based on local evidence, the framework provides a robust defense against communication sabotage without compromising efficiency. Future work will look toward scaling this approach to larger agent populations and extending the logic to continuous action spaces.

Summary updated

Abstract

Communication in multi-agent reinforcement learning (MARL) has been proven to effectively promote cooperation among agents recently. Since communication in real-world scenarios is vulnerable to noises and adversarial attacks, it is crucial to develop robust communicative MARL technique. However, existing research in this domain has predominantly focused on passive defense strategies, where agents receive all messages equally, making it hard to balance performance and robustness. We propose an active defense strategy, where agents automatically reduce the impact of potentially harmful messages on the final decision. There are two challenges to implement this strategy, that are defining unreliable messages and adjusting the unreliable messages' impact on the final decision properly. To address them, we design an Active Defense Multi-Agent Communication framework (ADMAC), which estimates the reliability of received messages and adjusts their impact on the final decision accordingly with the help of a decomposable decision structure. The superiority of ADMAC over existing methods is validated by experiments in three communication-critical tasks under four types of attacks.
BibTeX
@inproceedings{yu2024robust,
  author = {Lebin Yu and Yunbo Qiu and Quanming Yao and Yuan Shen and Xudong Zhang and Jian Wang},
  title = {{Robust Communicative Multi-Agent Reinforcement Learning with Active Defense}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29708},
}

Loading related content...

Figure