Abstract
Communication is a crucial factor for the multi-agent world to stay organized and productive. Recently, Deep Reinforcement Learning (DRL) has been applied to learn communication strategies and control policies for multiple agents. However, the practical limited bandwidth in multi-agent communication has been largely ignored by existing DRL methods. Specifically, many methods keep sending messages incessantly, which consumes too much bandwidth and makes them inapplicable to bandwidth-restricted systems. To handle this problem, we propose a gating mechanism to adaptively prune less beneficial messages. Experiments demonstrate that this mechanism can prune a significant portion of messages with little impact on performance. In fact, performance may be improved by pruning redundant messages. Moreover, the proposed gating mechanism is applicable to several previous methods, equipping them with the ability to address bandwidth-restricted settings.
Introduction
In multi-agent systems, coordination through communication is essential for achieving collective goals. However, most current Deep Reinforcement Learning (DRL) approaches operate under the assumption of unlimited communication bandwidth. This leads to agents "chattering" continuously, which is impractical in real-world scenarios where bandwidth is a scarce resource—such as in wireless sensor networks with limited battery life or network routers with fixed capacities.
The challenge lies in transitioning from "always-on" communication to selective communication. Agents must be able to decide in real-time whether a specific message provides enough benefit to justify its transmission cost. This paper addresses this by introducing a principled gating mechanism to prune redundant or low-value messages.
Methods
Actor-Critic Message Learner (ACML)
The authors first establish a baseline architecture called ACML. It integrates several successful components from prior research: a shared "Message Coordinator" (similar to CommNet) to facilitate information exchange and a centralized critic (inspired by MADDPG) to address the challenges of partial observability and non-stationarity in multi-agent environments.
Gated-ACML and the Gating Mechanism
The primary contribution is Gated-ACML, which augments each agent with a GatingNet. This network acts as a binary switch for outgoing messages.
- Decision Logic: For every potential message, the GatingNet outputs a probability $p$. If $p > 0.5$, the gate opens and the message is transmitted. Otherwise, the message is pruned and replaced with a zero vector.
- Auxiliary Task Training: Since the binary gate is non-differentiable, it is trained via an auxiliary task. The gate learns to predict whether the $\Delta Q$—the difference in the team's expected reward with versus without that specific message—exceeds a threshold $T$.
- Thresholding Strategies: The researchers evaluated both a fixed threshold (e.g., pruning the bottom 70% of messages by $\Delta Q$) and a dynamic threshold that adapts to environmental changes using an exponential moving average.
Experimental Setup
The proposed mechanism was tested across three distinct simulated environments:
- Traffic Control: Agents manage vehicle flow at an intersection, aiming to minimize delays and avoid collisions through coordination.
- Packet Routing: Routers coordinate to minimize Maximum Link Utilization (MLU) across a network topology.
- Wifi Access Point (AP) Configuration: APs coordinate transmission power to maximize user signal strength (RSSI) while minimizing inter-cell interference.
Key Results and Insights
The experiments yielded several significant findings:
- High Pruning Efficiency: In the Traffic Control task, Gated-ACML pruned 70% to 80% of messages while achieving higher rewards than the "always-talking" baseline. Messages were found to be critical only when vehicles were near the junction; pruning distant communications reduced "distraction" and improved driving efficiency.
- Performance Gains through Pruning: In packet routing, pruning 24% of messages actually increased the reward by 13.59%. This suggests the gating mechanism serves as a noise filter, removing low-signal messages that might otherwise confuse the agents' policies.
- Broad Applicability: The gating mechanism is modular. When integrated into existing models like CommNet and AMP, it successfully enabled them to operate effectively under bandwidth constraints.
- Robustness: Gated-ACML significantly outperformed ATOC, a prior state-of-the-art method, particularly in non-spatial tasks like routing where simple distance-based communication heuristics fail.
Conclusion and Future Work
This work demonstrates that intelligent message pruning can drastically reduce bandwidth consumption in multi-agent systems with minimal performance impact, and in some cases, even improve it by filtering out redundant information.
Future research will focus on improving the auxiliary task's performance in high-dimensional state spaces, such as complex routing scenarios where Q-value estimation becomes difficult. Additionally, exploring more scalable training methods beyond the current round-robin approach will be crucial for applying these techniques to systems with thousands of agents.
Summary updated