Introduction
Multi-agent reinforcement learning (MARL) has achieved significant milestones in domains like traffic control and competitive gaming. However, it often struggles with multistage tasks—sequential scenarios where agents must complete a series of objectives (e.g., Goal A, then B, then C). In these environments, the search space expands exponentially. Agents must not only learn how to interact but also determine the correct sequence of subtasks.
Existing MARL algorithms typically focus on agent-level communication, such as observing neighbors' actions. Without a shared understanding of the high-level objective, agents often engage in "aimless exploration," leading to poor sample efficiency and coordination failure. To address this, the Multi-Agent Goal ConsIstenCy (MAGIC) framework introduces a mechanism for agents to reach a consensus on immediate priorities before acting.
The MAGIC Framework
MAGIC incorporates "goal cognition" into the learning process using a goal-oriented actor-critic architecture. It relies on two primary innovations to foster teamwork:
- Goal Cognition Module (GCM): Agents utilize a soft-attention mechanism to weigh the importance of various sub-goals based on local observations. This allows each agent to "cognize" the current stage of the task.
- Two-Level Consistency Training: To prevent agents from working at cross-purposes, MAGIC employs two specific loss functions to align their internal states.
As shown in the framework diagram, the Self-Cognitive Dissonance (SCD-loss) aligns an individual agent’s local view with the centralized critic's global view, helping overcome partial observability. Simultaneously, the Team Cognitive Dissonance (TCD-loss) aligns goal distributions across the entire team, penalizing agents for pursuing different sub-goals at the same time. The system follows the Centralized Training and Decentralized Execution (CTDE) paradigm: agents share information during training to learn these consistencies but act independently during deployment.
Experimental Evaluation
The researchers evaluated MAGIC across three distinct simulated environments:
- Resource Collection: Agents mine multiple resource pools in a specific sequence. This task tests scalability as the number of agents increases.
- Multi-Point Transportation: Agents must cooperatively push a heavy ball to several destinations in order, requiring precise physical coordination and rapid strategy switching.
- Endangered Wildlife Rescue: A sparse-reward task where agents rescue animals based on injury severity. This environment tests coordination under high-pressure, "all-or-nothing" reward structures.
Key Results and Insights
MAGIC demonstrated superior performance across all benchmarks, particularly in terms of scalability and convergence speed. In the Resource Collection task, while standard baselines like MADDPG saw performance collapse as team size grew, MAGIC maintained high efficiency.
As illustrated in the training curves for a task with four agents and three sequential goals, MAGIC (blue line) reaches a near-perfect completion rate significantly faster than competing methods. Beyond raw metrics, the framework enables sophisticated behaviors; for instance, in transportation tasks, agents learned to move to the opposite side of the ball to act as a "brake" when approaching a destination.
Two critical insights emerged from this research:
- Goal-Level Coordination: By aligning on what they are trying to achieve, the how (the specific behavior) becomes much easier for agents to learn. This effectively prunes the combinatorial explosion of the search space.
- The Power of Consensus: Ablation studies showed that team alignment (TCD-loss) is often more vital than individual accuracy. Being "wrong together"—where the whole team pursues the same sub-optimal goal—is frequently more productive than agents pursuing different "correct" sub-goals simultaneously.
Conclusion and Future Work
While MAGIC effectively solves the aimless exploration problem for unified team goals, it currently drives the entire team toward a single objective. Future work will explore "sub-group consistency," allowing teams to dynamically split into clusters to tackle different sub-goals in parallel. Additionally, moving from pre-defined sub-goals to autonomous goal discovery from raw sensory input remains a key frontier for the framework.
Summary updated