Abstract
Multi-agent reinforcement learning (MARL) often struggles with the "curse of dimensionality" in complex environments. While subtask decomposition offers a solution, traditional hierarchical methods are often rigid and slow to adapt. This paper presents SMAUG (Sliding Multidimensional tAsk window-based mUti-agent reinforceMent learninG), a framework that utilizes a sliding multidimensional task window and an inference network to enable real-time subtask recognition and switching. Evaluated on StarCraft II (SMAC II), SMAUG demonstrates superior win rates, faster convergence, and enhanced stability compared to state-of-the-art baselines.
Introduction
In multi-agent reinforcement learning, the exponential growth of the joint action space as more agents are added makes effective cooperation difficult to achieve through brute-force exploration. Subtask decomposition—breaking a complex mission into smaller, manageable subtasks like kiting or healing—is a common remedy. However, most current methods rely on Hierarchical Reinforcement Learning (HRL), which often suffers from "stiffness." These models typically limit the number of subtasks or force agents to stick to a subtask for a fixed period, leading to sub-optimal performance in dynamic, fast-paced scenarios.
Methodology
SMAUG operates under the Centralized Training with Decentralized Execution (CTDE) paradigm and introduces several key innovations to break the rigidity of traditional HRL:
- Sliding Multidimensional Task Window: Instead of a fixed temporal snapshot, SMAUG uses a sliding window to extract information from trajectory segments of varying lengths. This captures both short-term micro-behaviors and long-term strategic trends.
- Inference Network: This "look-ahead" mechanism iteratively predicts future observations and rewards. By combining these predictions with past observations, the agent gains a clearer picture of its current trajectory and intended subtask.
- Subtask-Oriented Policy: The framework employs Gated Recurrent Units (GRUs) and multi-head attention to process these trajectory windows, allowing agents to attend to the most informative parts of their history and predicted future.
- Intrinsic Motivation Rewards: Using Mutual Information (MI), the model encourages diversity in trajectories for different subtasks, preventing "role collapse" where agents become redundant.
Experimental Results
The framework was evaluated using the StarCraft Multi-Agent Challenge (SMAC) II benchmark on several "Hard" and "Super Hard" maps, including MMM2 and corridor. SMAUG was compared against state-of-the-art baselines including QMIX, ROMA, QTRAN, COMA, and IQL.
- Superior Performance and Convergence: SMAUG outperformed all baselines in final win rates. As seen in the results for the 'corridor' map, SMAUG exhibits a "swift rise" in rewards, learning effective strategies much earlier in the training process than its competitors.
- High Reliability: The algorithm demonstrated significantly lower standard deviation across different training seeds, indicating higher stability and less sensitivity to initial conditions.
- Optimal Temporal Granularity: Ablation studies confirmed that a multidimensional window allows the model to see at different temporal resolutions simultaneously, which is crucial as different subtasks (e.g., micro-maneuvers vs. flanking) occur over different timescales.
Conclusion
SMAUG addresses the rigidity of traditional subtask decomposition in MARL by combining multi-scale temporal windows with predictive inference. This enables more fluid and effective multi-agent coordination in complex environments. While the framework shows impressive results, future work will focus on reducing the computational overhead of the iterative inference loop and exploring dynamic adjustment of the task window size based on environmental complexity.
Summary updated