Goal Consistency: An Effective Multi-Agent Cooperative Method for Multistage Tasks

Xinning Chen, Xuan Liu, Shigeng Zhang, Bo Ding, Kenli Li

Introduction

Multi-agent reinforcement learning (MARL) has achieved significant milestones in domains like traffic control and competitive gaming. However, it often struggles with multistage tasks—sequential scenarios where agents must complete a series of objectives (e.g., Goal A, then B, then C). In these environments, the search space expands exponentially. Agents must not only learn how to interact but also determine the correct sequence of subtasks.

Existing MARL algorithms typically focus on agent-level communication, such as observing neighbors' actions. Without a shared understanding of the high-level objective, agents often engage in "aimless exploration," leading to poor sample efficiency and coordination failure. To address this, the Multi-Agent Goal ConsIstenCy (MAGIC) framework introduces a mechanism for agents to reach a consensus on immediate priorities before acting.

The MAGIC Framework

MAGIC incorporates "goal cognition" into the learning process using a goal-oriented actor-critic architecture. It relies on two primary innovations to foster teamwork:

  1. Goal Cognition Module (GCM): Agents utilize a soft-attention mechanism to weigh the importance of various sub-goals based on local observations. This allows each agent to "cognize" the current stage of the task.
  2. Two-Level Consistency Training: To prevent agents from working at cross-purposes, MAGIC employs two specific loss functions to align their internal states.
The MAGIC framework's two-level consistency training aligns local goal cognition with global views (SCD-Loss) and team-wide consensus (TCD-Loss).

As shown in the framework diagram, the Self-Cognitive Dissonance (SCD-loss) aligns an individual agent’s local view with the centralized critic's global view, helping overcome partial observability. Simultaneously, the Team Cognitive Dissonance (TCD-loss) aligns goal distributions across the entire team, penalizing agents for pursuing different sub-goals at the same time. The system follows the Centralized Training and Decentralized Execution (CTDE) paradigm: agents share information during training to learn these consistencies but act independently during deployment.

Experimental Evaluation

The researchers evaluated MAGIC across three distinct simulated environments:

  • Resource Collection: Agents mine multiple resource pools in a specific sequence. This task tests scalability as the number of agents increases.
  • Multi-Point Transportation: Agents must cooperatively push a heavy ball to several destinations in order, requiring precise physical coordination and rapid strategy switching.
  • Endangered Wildlife Rescue: A sparse-reward task where agents rescue animals based on injury severity. This environment tests coordination under high-pressure, "all-or-nothing" reward structures.

Key Results and Insights

MAGIC demonstrated superior performance across all benchmarks, particularly in terms of scalability and convergence speed. In the Resource Collection task, while standard baselines like MADDPG saw performance collapse as team size grew, MAGIC maintained high efficiency.

Performance comparison in a multistage task (N=4, L=3), where MAGIC significantly outperforms standard MARL baselines in goal completion.

As illustrated in the training curves for a task with four agents and three sequential goals, MAGIC (blue line) reaches a near-perfect completion rate significantly faster than competing methods. Beyond raw metrics, the framework enables sophisticated behaviors; for instance, in transportation tasks, agents learned to move to the opposite side of the ball to act as a "brake" when approaching a destination.

Two critical insights emerged from this research:

  • Goal-Level Coordination: By aligning on what they are trying to achieve, the how (the specific behavior) becomes much easier for agents to learn. This effectively prunes the combinatorial explosion of the search space.
  • The Power of Consensus: Ablation studies showed that team alignment (TCD-loss) is often more vital than individual accuracy. Being "wrong together"—where the whole team pursues the same sub-optimal goal—is frequently more productive than agents pursuing different "correct" sub-goals simultaneously.

Conclusion and Future Work

While MAGIC effectively solves the aimless exploration problem for unified team goals, it currently drives the entire team toward a single objective. Future work will explore "sub-group consistency," allowing teams to dynamically split into clusters to tackle different sub-goals in parallel. Additionally, moving from pre-defined sub-goals to autonomous goal discovery from raw sensory input remains a key frontier for the framework.

Summary updated

Abstract

Although multistage tasks involving multiple sequential goals are common in real-world applications, they are not fully studied in multi-agent reinforcement learning (MARL). To accomplish a multi-stage task, agents have to achieve cooperation on different subtasks. Exploring the collaborative patterns of different subtasks and the sequence of completing the subtasks leads to an explosion in the search space, which poses great challenges to policy learning. Existing works designed for single-stage tasks where agents learn to cooperate only once usually suffer from low sample efficiency in multi-stage tasks as agents explore aimlessly. Inspired by human’s improving cooperation through goal consistency, we propose Multi-Agent Goal Consistency (MAGIC) framework to improve sample efficiency for learning in multi-stage tasks. MAGIC adopts a goal-oriented actor-critic model to learn both local and global views of goal cognition, which helps agents understand the task at the goal level so that they can conduct targeted exploration accordingly. Moreover, to improve exploration efficiency, MAGIC employs two-level goal consistency training to drive agents to formulate a consistent goal cognition. Experimental results show that MAGIC significantly improves sample efficiency and facilitates cooperation among agents compared with state-of-art MARL algorithms in several challenging multistage tasks.
BibTeX
@inproceedings{chen2022goal,
  author = {Xinning Chen and Xuan Liu and Shigeng Zhang and Bo Ding and Kenli Li},
  title = {{Goal Consistency: An Effective Multi-Agent Cooperative Method for Multistage Tasks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/25},
}
Figure