TL;DR
This paper introduces the Composite Task Challenge (CTC), a new benchmark for Multi-Agent Reinforcement Learning (MARL) that explicitly requires Division of Labor (DOL). The authors demonstrate that current state-of-the-art MARL algorithms fail completely on these tasks, and they propose e-QMIX—an architecture using Mixture of Experts—alongside specialized reward shaping to provide a first step toward solving them.
What Problem Does This Paper Address?
In both nature and human industry, cooperation thrives on the division of labor: different individuals specialize in different tasks to achieve a common goal. While many MARL researchers have tried to bake DOL into their algorithms, the benchmarks used to test them (like SMAC or MPE) often don't actually require it. In many existing tasks, agents can succeed by simply following the same policy (e.g., all agents "focus firing" on one enemy) or by having a few high-performing agents carry the rest of the team.
The authors argue that this "forgiving" nature of current benchmarks creates a gap between research and real-world applications. To bridge this, they created CTC, a suite of tasks where:
- DOL is mandatory: Subtasks are spatially separated and happen simultaneously, so a single agent cannot do everything.
- Failure is absolute: If even one atomic subtask fails, the entire mission is a failure.
Methods
The authors propose two main contributions to tackle the CTC benchmark:
- e-QMIX (Extended QMIX): A modified version of the popular QMIX algorithm designed to promote policy diversity while maintaining the efficiency of parameter sharing. It uses two sequential Mixture of Experts (MoE) modules:
- Feature Diversity: A sparse MoE that allows agents to extract different features from the same input.
- Output Diversity: A dense MoE that generates distinct action policies for different agents based on their unique observations.
- RER (Rule-based External Reward): To combat the "wandering issue"—where agents finish their subtask and then move aimlessly instead of helping others—the authors designed a reward signal that:
- Prioritizes high-value targets (like Medivacs).
- Encourages agents to remain engaged in offensive actions even after their primary objective is met.
Datasets
The paper introduces the Composite Task Challenge (CTC), implemented within the StarCraft Multi-Agent Challenge (SMAC) framework. It consists of four primary tasks:
- Defense_3_Subtask (D3S) & Defense_4_Subtask (D4S): Agents must defend multiple bases simultaneously against waves of enemies.
- Mixed_2_Subtask (M2S) & Mixed_3_Subtask (M3S): A combination of "Defense" and "Pursuit" (preventing enemies from escaping) subtasks.
The benchmark uses heterogeneous Terran units (Marines, Marauders, and Medivacs) to further test how algorithms assign specific roles to specific unit types.
Key Insights and Ideas
- The "Wandering Issue": A major discovery was that even when agents successfully specialized to complete a subtask, they often became "idle wanderers" afterward. In a system where every subtask must succeed, an agent that finishes early but fails to support a struggling teammate causes a total mission failure.
- Current SOTA is Insufficient: The most striking finding is that nine representative MARL methods (including RODE, ROMA, and QMIX) achieved a 0% win rate across all CTC tasks. This suggests that current "specialization" mechanisms in MARL are much weaker than previously thought.
- Strict Dependencies Expose Weakness: By enforcing a "fail-one, fail-all" rule, the authors successfully created a "discriminative" testbed that can clearly separate algorithms that truly understand coordination from those that just exploit environment rewards.
Key Results
- Baseline Failure: All 9 baseline methods failed to win a single game in the CTC suite.
- e-QMIX Success: When combined with the RER reward shaping, e-QMIX achieved non-zero win rates across all tasks, reaching as high as 81.25% on the Defense_3_Subtask.
- Ablation Findings: The authors found that "Output Diversity" (the second MoE layer) was more critical for success than "Feature Diversity." Removing the output diversity module often dropped the win rate back to zero.
Limitations and Future Work
- Manual Reward Shaping: The RER solution is rule-based and tailored specifically to the SMAC environment. It is not a "general" solution for DOL, and the authors admit it may not generalize to other types of tasks.
- Suboptimal Performance: Even the proposed e-QMIX solution struggles with the more complex tasks (like Mixed_3_Subtask), showing there is still a significant performance gap to be filled.
- Future Direction: The authors suggest that future research should focus on more automated, generalizable ways to learn DOL and inter-group cooperation without relying on hand-crafted rewards.
Summary updated