Abstract
The aim of multi-agent reinforcement learning systems is to provide interacting agents with the ability to collaboratively learn and adapt to the behavior of other agents. In many real-world applications, the agents can only acquire a partial view of the world. However, in realistic settings, one or more agents that show arbitrarily faulty or malicious behavior may suffice to let the current coordination mechanisms fail. In this paper, we study a practical scenario considering the security issues in the presence of agents with arbitrarily faulty or malicious behavior. Under these circumstances, learning an optimal policy becomes particularly challenging, even in the unrealistic case that an agent's policy can be made conditional upon all other agents' observations. To overcome these difficulties, we present an Attention-based Fault-Tolerant (FT-Attn) algorithm which selects correct and relevant information for each agent at every time-step. The multi-head attention mechanism enables the agents to learn effective communication policies through experience concurrently to the action policies. Empirical results have shown that FT-Attn beats previous state-of-the-art methods in some complex environments and can adapt to various kinds of noisy environments without tuning the complexity of the algorithm. Furthermore, FT-Attn can effectively deal with the complex situation where an agent needs to reach multiple agents' correct observation at the same time.
Introduction
In multi-agent reinforcement learning (MARL), agents typically share observations to build a comprehensive understanding of their environment. However, real-world deployments—such as drone swarms in search-and-rescue—often face sensor failures or malicious interference. Standard MARL algorithms assume shared data is reliable; if even one agent provides "garbage" data, the entire coordination mechanism can collapse.
The challenge lies in Information Filtering: agents must dynamically identify which neighbors provide relevant, correct information without prior knowledge of which agents are "gifted" with accurate sensors. Furthermore, any solution must handle varying agent counts and noise types without extensive manual tuning.
Methods: The FT-Attn Algorithm
To address these challenges, we propose the Attention-based Fault-Tolerant (FT-Attn) algorithm. The architecture consists of three primary modules designed to filter unreliable data:
- Encoder: Each agent $i$ processes its local observations $o_i$ and actions $a_i$ into an embedding $e_i$.
- Multi-head Attention Mechanism: This serves as a dynamic trust filter. Each agent generates a "query" to evaluate other agents' "keys," determining the "value" (weight) of their information. The multi-head structure allows an agent to focus on different subsets of peers for different contextual reasons simultaneously.
- Q-Network (Critic): The value function is estimated by combining the agent's own state with the weighted, filtered information from its peers.
The system is trained using Soft Actor-Critic (SAC), which maximizes both expected reward and policy entropy. This encourages exploration and prevents agents from converging on sub-optimal communication strategies early in training.
Experimental Setup
We evaluated FT-Attn in a modified Cooperative Navigation environment where $N$ agents must reach $N$ landmarks. Crucially, only one "gifted" agent can see the true landmark positions; others receive noisy data. We tested three scenarios:
- Fixed: The gifted agent remains constant.
- Alternating: The gifted agent changes every episode.
- Dynamic: The gifted agent changes within an episode based on spatial positioning.
Key Results and Discussion
FT-Attn was compared against several baselines, including MADDPG and the previous state-of-the-art, MADDPG-M.
As shown in the results table, FT-Attn significantly outperforms MADDPG-M in the more challenging "Alternating" and "Dynamic" scenarios. Notably, FT-Attn performs nearly as well as DDPG-OC, a theoretical upper-bound model where communication is manually optimized. This indicates that FT-Attn successfully discovers optimal communication strategies autonomously.
Insights
- Attention as Trust: The model uses attention to "mute" peers whose data does not correlate with reward prediction.
- Entropy Reduction: Visualization of "Attention Entropy" shows that as training progresses, agents successfully narrow their focus from a broad set of peers to the specific "gifted" sources.
- Zero-Tuning Adaptation: The model demonstrated robustness across different noise levels and agent configurations without hyperparameter adjustments.
Conclusion and Future Work
FT-Attn provides a robust framework for fault-tolerant MARL by leveraging multi-head attention to aggregate information from multiple correct sources while ignoring noise. While effective for small swarms ($N=3$ to $5$), future research will explore scalability to hundreds of agents and performance in competitive settings. Additionally, the computational overhead of multi-head attention in hardware-constrained robotics remains a key area for optimization.
Summary updated