Efficient Multi-Agent Communication via Shapley Message Value

Di Xue, Lei Yuan, Zongzhang Zhang, Yang YuIJCAI 2022 · OpenReview

Abstract

Utilizing messages from teammates is crucial in cooperative multi-agent tasks due to the partially observable nature of the environment. Naively asking messages from all teammates without pruning may confuse individual agents, hindering the learning process and impairing the whole system's performance. Most previous work either utilizes a gate or employs an attention mechanism to extract relatively important messages. However, they do not explicitly evaluate each message's value, failing to learn an efficient communication protocol in more complex scenarios. To tackle this issue, we model the teammates of an agent as a message coalition and calculate the Shapley Message Value (SMV) of each agent within it. SMV reflects the contribution of each message to an agent and redundant messages can be spotted in this way effectively. On top of that, we design a novel framework named Shapley Message Selector (SMS), which learns to predict the SMVs of teammates for an agent solely based on local information so that the agent can only query those teammates with positive SMVs. Empirically, we demonstrate that our method can prune redundant messages and achieve comparable or better performance in various multi-agent cooperative scenarios than full communication settings and existing strong baselines.

Introduction

In cooperative Multi-Agent Reinforcement Learning (MARL), agents often operate in partially observable environments. Sharing information with teammates is essential for success, but a naive broadcast approach—where every agent communicates with every other agent—quickly leads to information overload as the number of agents increases. This "noise" from irrelevant or redundant messages can confuse individual agents, leading to sub-optimal policies and slow convergence.

While previous research has explored attention mechanisms and gating to filter communication, these methods often treat messages as black boxes. They lack an explicit evaluation of a message's actual value to the receiver. This paper addresses this gap by introducing the Shapley Message Value (SMV), a mathematically rigorous way to quantify the marginal contribution of each message using cooperative game theory.

Methodology: The SMS Framework

The proposed framework, Shapley Message Selector (SMS), explicitly evaluates message contributions during training and uses these evaluations to prune communication during execution.

Overview of the SMS framework showing the communication-based policy (a) and the linearly decomposable critic (b).

The architecture consists of two primary components:

  1. Linearly Decomposable Critic: Under the Centralized Training with Decentralized Execution (CTDE) paradigm, a centralized critic decomposes the global reward into individual agent utilities. This allows the system to isolate the value generated by specific agents.
  2. Shapley Message Value (SMV): By modeling teammates as a "message coalition," the authors calculate the SMV to measure how much a teammate's message improves a receiving agent's expected utility. If a message improves expected utility, it has a positive SMV.
  3. Shapley Message Selector (SMS): Since exact SMVs require global state information unavailable during decentralized execution, a local neural network (the SMS) is trained via supervised learning to predict these values from local observations. During execution, agents only request messages from teammates predicted to have a positive SMV.

Experimental Evaluation

The method was evaluated across three distinct environments:

  • Listener-Speaker: A custom grid-world designed to test redundancy pruning, where listeners must find specific paired speakers among irrelevant ones.
  • Hallway: A cooperative navigation task requiring synchronization under partial observability.
  • StarCraft II Micromanagement Benchmark (SMAC): A standard, high-complexity MARL benchmark testing asymmetric and difficult maps.

Key Results and Insights

The SMS framework demonstrates that "more communication" is not always better. In the Listener-Speaker task, SMS achieved near-optimal communication rates, whereas "Full Communication" settings actually harmed performance due to overfitting on redundant noise.

Performance comparison on the SMAC benchmark, showing SMS (ours) achieving higher win rates and faster convergence than strong baselines.

On the challenging SMAC maps, SMS consistently outperformed strong communication baselines like MAGIC, NDQ, and TarMAC, as well as non-communication methods like QMIX. The results indicate that explicitly valuing messages allows for more efficient protocol learning and better overall system performance. Visualizations of the Hallway task further showed that agents learned logical communication patterns, only requesting messages when synchronization was critical.

Conclusion

By treating incoming messages as players in a cooperative game, the Shapley value provides a fair and interpretable method for credit assignment in multi-agent communication. The SMS framework successfully distills global value information into local selectors, enabling smart, decentralized communication that prunes redundancy and improves learning stability in complex cooperative tasks.

Summary updated

Abstract

Utilizing messages from teammates is crucial in cooperative multi-agent tasks due to the partially observable nature of the environment. Naively asking messages from all teammates without pruning may confuse individual agents, hindering the learning process and impairing the whole system's performance. Most previous work either utilizes a gate or employs an attention mechanism to extract relatively important messages. However, they do not explicitly evaluate each message's value, failing to learn an efficient communication protocol in more complex scenarios. To tackle this issue, we model the teammates of an agent as a message coalition and calculate the Shapley Message Value (SMV) of each agent within it. SMV reflects the contribution of each message to an agent and redundant messages can be spotted in this way effectively. On top of that, we design a novel framework named Shapley Message Selector (SMS), which learns to predict the SMVs of teammates for an agent solely based on local information so that the agent can only query those teammates with positive SMVs. Empirically, we demonstrate that our method can prune redundant messages and achieve comparable or better performance in various multi-agent cooperative scenarios than full communication settings and existing strong baselines.
BibTeX
@inproceedings{xue2022efficient,
  author = {Di Xue and Lei Yuan and Zongzhang Zhang and Yang Yu},
  title = {{Efficient Multi-Agent Communication via Shapley Message Value}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=IAYDYK3YQZ},
}
Figure