Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learning

Yue Jin, Shuangqing Wei, Jian Yuan, Xudong Zhang

Abstract

Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is proposed to improve MARL. Specifically, by distinguishing the hierarchy of coordination, we propose to enhance decision coordination at meso level with an advisor and leverage a policy discriminator to advise agents' learning at micro level. The advisor learns to aggregate decision information in both spatial and temporal domains and generates coordinated decisions by employing a spatiotemporal dual graph convolutional neural network with a task-oriented objective function. Each agent learns from the advice via a policy generative adversarial learning method where a discriminator distinguishes between the policies of the agent and the advisor and boosts both of them based on its judgement. Experimental results indicate the advantage of LALA over baseline approaches in terms of both learning efficiency and coordination capability. Coordination mechanism is investigated from the perspective of multilevel emergence dynamics and mutual information point of view, which provides a novel perspective and method to analyze and improve MARL algorithms.

Introduction

In Multi-Agent Reinforcement Learning (MARL), teaching agents to cooperate—such as avoiding collisions or dividing up targets—is a daunting challenge. While previous research has tackled this through communication channels, credit assignment, or expert demonstrations, these methods often overlook the intrinsic spatiotemporal structure of the agents' decisions.

This paper argues that coordination should be viewed through the lens of "Multilevel Emergence Dynamics" (MED). In standard MARL, systems often attempt to jump directly from micro-level individual behaviors to macro-level task success. LALA addresses the missing "meso-level" interplay—a mechanism that explicitly examines how decisions interact across space (between agents) and time (across sequential steps) to actively resolve conflicts during the learning process.

The LALA Framework

To inject this meso-level coordination, the LALA framework operates under a Centralized Training, Decentralized Execution (CTDE) paradigm. It consists of two main components:

  1. The Advisor (DualGCN): During training, a centralized advisor observes the decisions of all agents. It constructs a spatiotemporal graph where spatial edges connect different agents at the same time step (to model conflicts) and temporal edges connect the same agent across adjacent time steps (to model continuity). A Spatiotemporal Dual Graph Convolutional Network (DualGCN) processes this graph to generate "advice"—coordinated, conflict-free decisions.
  2. The Learner (PLGAN): Agents learn from this advice using a Policy-Level Generative Adversarial Network. A Transformer-based discriminator analyzes sets of state-action pairs to distinguish between raw agent policies and smoothed advisor policies. Agents update their policies to "fool" the discriminator, pulling their behavior toward the advisor's coordinated strategy while still exploring the environment for rewards.
The LALA framework architecture, featuring the centralized Advisor (DualGCN) and the Discriminator-based Learner (PLGAN).

Experimental Evaluation

The authors evaluate LALA on a simulated Cooperative Navigation Task. In this setup, $N$ agents must navigate a continuous $15 \times 15$ environment to reach $N$ discrete targets in the minimum amount of time. Agents are penalized for time spent and for "conflicts" (selecting the same target). The environment is tested with varying complexities of $N = 7, 8, 9,$ and $10$ agents/targets.

Results and Discussion

LALA was compared against a standard MARL baseline (No-Advice) and several ablation baselines, including Knowledge Distillation Advising (KDA).

As the number of agents scales, LALA's advantage becomes stark. In the 10-agent scenario ($N=10$), LALA achieved a 75.7% cooperation success rate, while the standard No-Advice baseline achieved only 1.5%. Furthermore, LALA achieved the lowest normalized navigation time across all agent counts, indicating that agents resolved conflicts and reached targets much faster than with other methods.

Comparison of normalized navigation time for N=7 and N=10 agents, showing LALA's superior performance and faster convergence as complexity increases.

Beyond performance metrics, the authors used Mutual Information Neural Estimation (MINE) to measure how well an agent's local perception predicts other agents' actions. They found that LALA significantly increases this mutual information, proving that the centralized advice actively helps decentralized agents "understand" each other better.

Conclusion

By explicitly building a meso-level module to handle inter-agent decision structures, LALA enables macro-level task success to emerge more efficiently. While currently tested on cooperative navigation, the framework provides a structured way to think about coordination in MARL. Future work will focus on verifying LALA across a wider variety of heterogeneous and competitive benchmarks, as well as developing more powerful Graph Convolutional Networks to further unlock the potential of meso-level coordination.

Summary updated

Abstract

Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is proposed to improve MARL. Specifically, by distinguishing the hierarchy of coordination, we propose to enhance decision coordination at meso level with an advisor and leverage a policy discriminator to advise agents' learning at micro level. The advisor learns to aggregate decision information in both spatial and temporal domains and generates coordinated decisions by employing a spatiotemporal dual graph convolutional neural network with a task-oriented objective function. Each agent learns from the advice via a policy generative adversarial learning method where a discriminator distinguishes between the policies of the agent and the advisor and boosts both of them based on its judgement. Experimental results indicate the advantage of LALA over baseline approaches in terms of both learning efficiency and coordination capability. Coordination mechanism is investigated from the perspective of multilevel emergence dynamics and mutual information point of view, which provides a novel perspective and method to analyze and improve MARL algorithms.
BibTeX
@inproceedings{jin2022learning,
  author = {Yue Jin and Shuangqing Wei and Jian Yuan and Xudong Zhang},
  title = {{Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learning}},
  booktitle = {AAMAS},
  year = {2022},
  url = {https://openreview.net/forum?id=oLmew32NgA},
}
Figure