Learning Agent Communication under Limited Bandwidth by Message Pruning

Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, Yan Ni

Abstract

Communication is a crucial factor for the multi-agent world to stay organized and productive. Recently, Deep Reinforcement Learning (DRL) has been applied to learn communication strategies and control policies for multiple agents. However, the practical limited bandwidth in multi-agent communication has been largely ignored by existing DRL methods. Specifically, many methods keep sending messages incessantly, which consumes too much bandwidth and makes them inapplicable to bandwidth-restricted systems. To handle this problem, we propose a gating mechanism to adaptively prune less beneficial messages. Experiments demonstrate that this mechanism can prune a significant portion of messages with little impact on performance. In fact, performance may be improved by pruning redundant messages. Moreover, the proposed gating mechanism is applicable to several previous methods, equipping them with the ability to address bandwidth-restricted settings.

Introduction

In multi-agent systems, coordination through communication is essential for achieving collective goals. However, most current Deep Reinforcement Learning (DRL) approaches operate under the assumption of unlimited communication bandwidth. This leads to agents "chattering" continuously, which is impractical in real-world scenarios where bandwidth is a scarce resource—such as in wireless sensor networks with limited battery life or network routers with fixed capacities.

The challenge lies in transitioning from "always-on" communication to selective communication. Agents must be able to decide in real-time whether a specific message provides enough benefit to justify its transmission cost. This paper addresses this by introducing a principled gating mechanism to prune redundant or low-value messages.

Methods

Actor-Critic Message Learner (ACML)

The authors first establish a baseline architecture called ACML. It integrates several successful components from prior research: a shared "Message Coordinator" (similar to CommNet) to facilitate information exchange and a centralized critic (inspired by MADDPG) to address the challenges of partial observability and non-stationarity in multi-agent environments.

The ACML architecture featuring ActorNet, CriticNet, and Message Coordinator components.

Gated-ACML and the Gating Mechanism

The primary contribution is Gated-ACML, which augments each agent with a GatingNet. This network acts as a binary switch for outgoing messages.

The GatingNet mechanism, which uses an auxiliary task to decide whether to prune messages.
  • Decision Logic: For every potential message, the GatingNet outputs a probability $p$. If $p > 0.5$, the gate opens and the message is transmitted. Otherwise, the message is pruned and replaced with a zero vector.
  • Auxiliary Task Training: Since the binary gate is non-differentiable, it is trained via an auxiliary task. The gate learns to predict whether the $\Delta Q$—the difference in the team's expected reward with versus without that specific message—exceeds a threshold $T$.
  • Thresholding Strategies: The researchers evaluated both a fixed threshold (e.g., pruning the bottom 70% of messages by $\Delta Q$) and a dynamic threshold that adapts to environmental changes using an exponential moving average.

Experimental Setup

The proposed mechanism was tested across three distinct simulated environments:

  1. Traffic Control: Agents manage vehicle flow at an intersection, aiming to minimize delays and avoid collisions through coordination.
  2. Packet Routing: Routers coordinate to minimize Maximum Link Utilization (MLU) across a network topology.
  3. Wifi Access Point (AP) Configuration: APs coordinate transmission power to maximize user signal strength (RSSI) while minimizing inter-cell interference.

Key Results and Insights

The experiments yielded several significant findings:

  • High Pruning Efficiency: In the Traffic Control task, Gated-ACML pruned 70% to 80% of messages while achieving higher rewards than the "always-talking" baseline. Messages were found to be critical only when vehicles were near the junction; pruning distant communications reduced "distraction" and improved driving efficiency.
  • Performance Gains through Pruning: In packet routing, pruning 24% of messages actually increased the reward by 13.59%. This suggests the gating mechanism serves as a noise filter, removing low-signal messages that might otherwise confuse the agents' policies.
  • Broad Applicability: The gating mechanism is modular. When integrated into existing models like CommNet and AMP, it successfully enabled them to operate effectively under bandwidth constraints.
  • Robustness: Gated-ACML significantly outperformed ATOC, a prior state-of-the-art method, particularly in non-spatial tasks like routing where simple distance-based communication heuristics fail.

Conclusion and Future Work

This work demonstrates that intelligent message pruning can drastically reduce bandwidth consumption in multi-agent systems with minimal performance impact, and in some cases, even improve it by filtering out redundant information.

Future research will focus on improving the auxiliary task's performance in high-dimensional state spaces, such as complex routing scenarios where Q-value estimation becomes difficult. Additionally, exploring more scalable training methods beyond the current round-robin approach will be crucial for applying these techniques to systems with thousands of agents.

Summary updated

Abstract

Communication is a crucial factor for the big multi-agent world to stay organized and productive. Recently, Deep Reinforcement Learning (DRL) has been applied to learn the communication strategy and the control policy for multiple agents. However, the practical limited bandwidth in multi-agent communication has been largely ignored by the existing DRL methods. Specifically, many methods keep sending messages incessantly, which consumes too much bandwidth. As a result, they are inapplicable to multi-agent systems with limited bandwidth. To handle this problem, we propose a gating mechanism to adaptively prune less beneficial messages. We evaluate the gating mechanism on several tasks. Experiments demonstrate that it can prune a lot of messages with little impact on performance. In fact, the performance may be greatly improved by pruning redundant messages. Moreover, the proposed gating mechanism is applicable to several previous methods, equipping them the ability to address bandwidth restricted settings.
BibTeX
@inproceedings{mao2020learning,
  author = {Hangyu Mao and Zhengchao Zhang and Zhen Xiao and Zhibo Gong and Yan Ni},
  title = {{Learning Agent Communication under Limited Bandwidth by Message Pruning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=RkqYVmOlV0E},
}
Figure