Promoting Resilience in Multi-Agent Reinforcement Learning via Confusion-Based Communication

Ofir Abu, Matthias Gerstgrasser, Jeffrey S. Rosenschein, Sarah Keren

Abstract

AI agents need to be robust to unexpected changes in their environment in order to safely operate in real-world scenarios. While some work has been done on this type of robustness in the single-agent case, in this work we introduce the idea that collaboration with other agents can help agents adapt to environment perturbations in multi-agent reinforcement learning (MARL) settings. We first formalize this notion of resilience of a group of agents. We then empirically evaluate different collaboration protocols and examine their effect on resilience. We see that all of the collaboration approaches considered lead to greater resilience compared to baseline, in line with our hypothesis. We discuss future direction and the general relevance of the concept of resilience introduced in this work.

Introduction

Most Reinforcement Learning (RL) research focuses on training agents to master static environments. However, real-world applications—from warehouse robotics to autonomous traffic management—are rarely stationary. A system must be able to handle "shocks," such as sudden obstacles or resource depletion. While resilience has been explored in single-agent settings, Multi-Agent Reinforcement Learning (MARL) introduces the "moving target" problem: agents must adapt not only to environmental shifts but also to the changing behaviors of their peers.

This paper addresses the lack of a unified framework for measuring how a team of agents recovers from unexpected perturbations. We hypothesize that collaboration is the primary driver of group resilience, allowing agents to pool information and adapt faster than they could in isolation.

Formalizing Group Resilience

To quantify a team's ability to bounce back, we introduce the concept of $C_K$-resilience. We define resilience as the ability of a group to maintain at least a fixed fraction ($C_K$) of its original performance when the environment undergoes a perturbation of magnitude $K$.

Conceptual representation of the environment space and the resilience region.

As illustrated in the conceptual framework above, we consider an environment $M$ within a broader manifold of possible environments $\mathcal{M}$. A perturbation shifts the system to a new state $M'$. Resilience is measured by comparing the group's post-perturbation performance against three potential benchmarks: a theoretical optimum, the group's own pre-perturbation baseline, or a relative recovery curve. This formalization allows us to rigorously test how different communication strategies impact the stability of the system.

Confusion-Based Communication Protocols

We propose that the most efficient way to promote resilience is through "confusion-based" communication. Rather than broadcasting all data, which is bandwidth-intensive and often redundant, agents should share experiences that are "surprising"—specifically, transitions where their internal models fail to predict the outcome (high TD-error).

We evaluated three primary protocols:

  1. Mandatory Broadcast: Agents are required to share their most "confusing" transitions with the group.
  2. Emergent Communication: Agents learn to send discrete symbols. We compare Self-Centric agents (minimizing their own confusion) with Global-Centric agents (rewarded for reducing the collective confusion of the group).
  3. suPER: A state-of-the-art protocol that uses high-error experiences to synchronize and update the internal world models of all agents simultaneously.

Experimental Evaluation

We tested these protocols across four MARL benchmarks: Cleanup (resource management), Harvest (tragedy-of-the-commons), Multi-Taxi (navigation), and a custom Coop-Mining domain. In Coop-Mining, agents must synchronize to mine high-value gold, making them highly sensitive to environmental "cave-ins."

Illustration of environmental perturbations in a grid-world domain, showing (a) the base state and (b) the state after obstacles are added.

Perturbations were introduced by modifying the environment mid-task, such as adding walls or changing resource regeneration rates (as seen in the figure above). We then measured the $C_K$-resilience of the group as they attempted to recover their utility.

Key Insights and Results

Our empirical results strongly support the hypothesis that collaboration acts as a safety net for MARL systems:

  • Collaboration Boosts Recovery: In all domains, communicative agents recovered a significantly larger share of their performance than "loner" agents. In the Cleanup domain, collaborative agents showed a 180% increase in resilience over non-communicative baselines during large-magnitude perturbations.
  • Global-Centricity is Key: In emergent communication, agents rewarded for reducing group confusion were far more resilient than those focused on individual confusion. This suggests that resilience is an emergent property of collective information sharing.
  • Efficiency of Surprise: Sharing only "misaligned" or surprising data points proved highly effective. This allows the group to update their world models rapidly without overwhelming communication channels.
  • Stability vs. Raw Performance: While some protocols like "Mandatory Communication" achieved high raw utility, "Emergent Global-Centric" protocols often provided more stable resilience scores across varying levels of environmental chaos.

Conclusion and Future Work

This work demonstrates that collaboration is essential for building resilient multi-agent systems. By sharing "confusing" experiences, agents can collectively navigate environmental shocks that would paralyze isolated individuals.

Future research will expand this framework to heterogeneous agents (e.g., different types of robots working together) and explore more sophisticated distance metrics for perturbations, such as Wasserstein distance. Additionally, investigating resilience in adversarial settings, where the environment or other agents may actively attempt to sabotage the group, remains a critical frontier for safe AI deployment.

Summary updated

Abstract

AI agents need to be robust to unexpected changes in their environment in order to safely operate in real-world scenarios. While some work has been done on this type of robustness in the single-agent case, in this work we introduce the idea that collaboration with other agents can help agents adapt to environment perturbations in multi-agent reinforcement learning settings. We first formalize this notion of resilience of a group of agents. We then empirically evaluate different collaboration protocols and examine their effect on resilience. We see that all of the collaboration approaches considered lead to greater resilience compared to baseline, in line with our hypothesis. We discuss future direction and the general relevance of the concept of resilience introduced in this work.
BibTeX
@misc{abu2021promoting,
  author = {Ofir Abu and Matthias Gerstgrasser and Jeffrey S. Rosenschein and Sarah Keren},
  title = {{Promoting Resilience in Multi-Agent Reinforcement Learning via Confusion-Based Communication}},
  year = {2021},
  eprint = {2111.06614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.06614},
}
Figure