Learning to Collaborate by Grouping: A Consensus-Oriented Strategy for Multi-Agent Reinforcement Learning

Jingqing Ruan, Xiaotian Hao, Dong Li, Hangyu Mao

Introduction

In multi-agent systems—ranging from drone swarms to autonomous warehouse robots—the primary challenge is coordination. While Multi-Agent Reinforcement Learning (MARL) has significantly improved individual agent policies, achieving cohesive teamwork remains difficult. Agents often lack a shared "group-level" strategy, leading to fragmented efforts rather than a unified approach.

The Consensus-oriented Strategy (CoS) framework addresses this by enabling agents to dynamically group themselves around shared objectives. By combining discrete latent embeddings with a hyper-network architecture, CoS balances high-level strategic consensus with the precise individual decision-making required for complex, stochastic environments.

The Coordination Challenge

Most current MARL algorithms focus on optimizing individual policies. While this makes agents "smarter" in isolation, it often results in weak collective behavior. Previous attempts to implement "roles" or "sub-tasks" have struggled with two main issues:

  1. Representation Instability: Group or role representations often fluctuate too rapidly during training, preventing agents from learning stable, effective behaviors.
  2. Guidance Deficit: There is often no clear mechanism to translate a high-level group goal into specific, refined actions for an individual agent based on its local context.

CoS proposes that effective teamwork requires a division of labor where agents reach a consensus on sub-goals while maintaining the flexibility to make rational decisions based on local observations.

The CoS Framework

The CoS framework introduces three primary technical innovations to bridge the gap between group strategy and individual action:

1. Vector Quantized Group Consensus (VQGC)

Drawing inspiration from Vector Quantized Variational Autoencoders (VQ-VAEs), CoS extracts discrete latent embeddings to represent group consensus. Unlike continuous embeddings, which can be noisy and unstable, these discrete codes act as a "stable anchor" for agents. To further distinguish these groups, the authors apply a hyperbolic space constraint using the Poincaré ball model. This geometric approach helps push different group representations apart in the latent space, making the division of labor more explicit.

2. Group Consensus Policy (GCP) via Hyper-networks

At the strategic level, CoS employs a hyper-network. Rather than treating the group embedding as a simple input feature, the hyper-network generates the actual weights for the base policy layers. This allows the group consensus to exert a powerful, top-down influence on agent behavior, effectively "reprogramming" the agent's policy based on the current group strategy.

3. Group-Guided Policy (GGP)

At the execution level, the group embedding serves as a "context prompt" for the agent’s individual observations. This ensures that while the agent follows the group's "big picture" plan, it can still refine its actions based on immediate, local details—much like a soccer player follows a coach's formation but decides how to dribble based on the defender immediately in front of them.

To ensure training stability, the authors also utilize Jump-start Dynamic Integration (JDI), a warm-up phase where individual policies and grouping logic are trained before fully integrating the top-down hyper-network influence.

Experimental Evaluation

The researchers evaluated CoS against state-of-the-art (SOTA) baselines like MAPPO, QMIX, and ROMA across three challenging benchmarks:

  • Discrete and Continuous Cooperative Navigation (d-CN/c-CN): Tasks requiring agents to coordinate movement to occupy specific landmarks.
  • Google Research Football (GRF): A complex environment where agents control a soccer team in scenarios such as 3vs1 and counterattack (easy and hard).

Key Results

  • Superior Performance: In Google Research Football, CoS consistently achieved higher win rates and faster completion times than all baselines.
  • Efficiency: In navigation tasks, CoS reached targets in fewer steps. For instance, in the 4-agent d-CN task, CoS averaged 19.8 steps, outperforming MAPPO's 21.2 steps.
  • Semantic Stability: T-SNE visualizations confirmed that CoS successfully learned to cluster agents into distinct, stable groups corresponding to functional roles within the environment. The use of discrete codes effectively filtered out "semantic noise," providing a clearer signal for coordination.

Conclusion and Future Work

CoS demonstrates that discrete, hyperbolic group representations combined with hyper-networks can significantly enhance multi-agent coordination. While the model showed limitations in extremely high-dimensional spaces (e.g., 10 agents in continuous navigation), it sets a new benchmark for structured teamwork in MARL.

Future research will explore universal group embeddings. By developing grouping knowledge that is not task-specific, agents could potentially transfer "teamwork logic"—such as forming defensive lines or search grids—across entirely different environments.

Summary updated

Abstract

Multi-agent systems require effective coordination between groups and individuals to achieve common goals. However, current multi-agent reinforcement learning (MARL) methods primarily focus on improving individual policies and do not adequately address group-level policies, which leads to weak cooperation. To address this issue, we propose a novel Consensus-oriented Strategy (CoS) that emphasizes group and individual policies simultaneously. Specifically, CoS comprises two main components: (a) the vector quantized group consensus module, which extracts discrete latent embeddings that represent the stable and discriminative group consensus, and (b) the group consensus-oriented strategy, which integrates the group policy using a hypernet and the individual policies using the group consensus, thereby promoting coordination at both the group and individual levels. Through empirical experiments on cooperative navigation tasks with both discrete and continuous spaces, as well as google research football, we demonstrate that CoS outperforms state-of-the-art MARL algorithms and achieves better collaboration, thus providing a promising solution for achieving effective coordination in multi-agent systems.
BibTeX
@inproceedings{ruan2023learning,
  author = {Jingqing Ruan and Xiaotian Hao and Dong Li and Hangyu Mao},
  title = {{Learning to Collaborate by Grouping: A Consensus-Oriented Strategy for Multi-Agent Reinforcement Learning}},
  booktitle = {ECAI},
  year = {2023},
  url = {https://openreview.net/forum?id=XZElSUKmb1},
}
Figure