Abstract
Multi-Agent Reinforcement Learning (MARL) often struggles with the "sight range dilemma," where giving agents too much or too little environmental visibility hurts their ability to learn. This paper introduces Dynamic Sight Range Selection (DSR), a plug-and-play method that uses a bandit algorithm to dynamically adjust agents' sight ranges during training. DSR seamlessly integrates with existing MARL algorithms, accelerating training and significantly boosting final performance across multiple benchmark environments.
Introduction: The Sight Range Dilemma
In real-world applications such as autonomous driving or warehouse robotics, agents operate under partial observability constrained by sensor limitations, known as the "sight range." A common assumption is that maximizing sight range is always beneficial. However, this research identifies a critical sight range dilemma:
- Insufficient Information: Small sight ranges leave agents blind to essential context, preventing effective coordination.
- Information Overload: Excessive sight ranges flood neural networks with irrelevant noise, slowing convergence and degrading final performance.
Finding the optimal "Goldilocks" sight range typically requires extensive manual tuning. While complex communication or attention mechanisms exist, they often rely on impractical global information. This paper proposes an automated approach to find optimal sight ranges using only local observations.
Methodology: Dynamic Sight Range Selection (DSR)
The authors propose Dynamic Sight Range Selection (DSR), a hierarchical framework that can be wrapped around any standard MARL algorithm.
Instead of a fixed sight range, DSR employs a meta-controller that selects a sight range $d$ from a discrete set at the start of each episode. The selection process utilizes the Sliding-Window Upper Confidence Bound (SW-UCB) algorithm. SW-UCB effectively balances exploration of new ranges with exploitation of high-reward ones. Crucially, the sliding-window mechanism allows the meta-controller to adapt to the non-stationary nature of learning agents, updating its preferences as agents become more proficient.
Experimental Evaluation
DSR was evaluated across three major cooperative MARL benchmarks: Level-Based Foraging (LBF), Multi-Robot Warehouse (RWARE), and the StarCraft Multi-Agent Challenge (SMAC/SMAC-DT).
Key Results
- Performance Gains: In complex LBF tasks, DSR achieved a mean return of 0.798, compared to 0.338 for the baseline. In SMAC, win rates improved from 29.2% to 71.2%.
- Training Acceleration: DSR consistently reached optimal performance in fewer training steps than fixed-range baselines.
- Curriculum Discovery: The meta-controller naturally discovered a learning curriculum, often selecting smaller ranges early in training and expanding them as agents required more context for advanced strategies.
- Algorithm Agnosticism: DSR successfully enhanced various algorithms, including QMIX, IQL, VDN, IPPO, and MAPPO.
Discussion and Conclusion
DSR demonstrates that more data is not always better; filtering irrelevant spatial information can significantly improve learning efficiency. Beyond performance, DSR offers interpretability for hardware design, indicating the minimum spatial information required for specific tasks.
Current limitations include the use of homogeneous sight ranges for all agents and the requirement for a predefined discrete set of options. Future research will explore heterogeneous sight range selection and adaptation to continuous observation spaces.
Summary updated