Offline Multi-Agent Reinforcement Learning with Knowledge Distillation

Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, Phillip IsolaNeurIPS 2022 · OpenReview

Training multiple agents to cooperate in complex environments is a cornerstone of modern AI. However, the standard "online" approach—where agents learn by trial and error in real-time—is often too expensive or dangerous for real-world applications. Imagine training a fleet of self-driving cars by letting them crash into each other until they learn to navigate a highway safely.

Offline Multi-Agent Reinforcement Learning (MARL) offers a safer, more practical solution: learning from a pre-collected dataset of experiences. Yet, traditional offline MARL methods struggle with "credit assignment"—figuring out which agent's action led to a specific reward—especially when they cannot interact with the environment to test their hypotheses.

In this paper, researchers propose a new framework that treats MARL as a sequence modeling problem, using a "Teacher-Student" architecture to bridge the gap between centralized coordination and decentralized execution.

The Challenge: TD Learning and CTDE

Most offline MARL algorithms rely on Temporal Difference (TD) learning, which uses "bootstrapping" (estimating future values based on current estimates). In multi-agent settings, this becomes incredibly complex because the environment's state changes based on the joint actions of all agents.

Furthermore, there is a fundamental tension in MARL known as Centralized Training vs. Decentralized Execution (CTDE). We want agents to learn from the "big picture" (what everyone is doing), but at test time, an agent usually only has access to its own local observations. Existing methods often struggle to transfer the complex coordination learned during centralized training into independent agents without losing performance.

A Sequence Modeling Approach

To overcome the limitations of TD-learning, the authors adopt the Decision Transformer (DT) architecture, which treats reinforcement learning as a sequence prediction task: given past states and desired rewards, predict the next action.

Comparison of MARL architectures, highlighting the proposed Teacher-Student distillation method.

Their framework consists of two main stages:

  1. The Teacher (Centralized): First, a "Teacher" Decision Transformer is trained with access to the observations, actions, and rewards of all agents. This global view allows the teacher to identify and recombine good behaviors hidden within suboptimal datasets.
  2. The Students (Decentralized): Next, individual "Student" policies are trained for each agent. These students only see their own local data, adhering to the decentralized execution requirement.

Structural Relation Distillation

The core innovation lies in how knowledge is transferred from the teacher to the students. Instead of just telling the students to mimic the teacher's raw actions (conventional distillation), the authors force the students to mimic the structural relations between agents.

Relational Policy Distillation focuses on matching the structural relationships between agents' features rather than just individual features.

They use a novel loss function that compares the "angles" (cosine similarity) between agents' feature representations in the teacher model versus the student models. This ensures that even if students have limited views, they maintain the internal "logic" and relational dynamics of the team's coordination.

Experimental Validation

The researchers evaluated their method across a diverse set of environments, including toy tasks (grid exploration), navigation, robotics/control (a multi-car traffic simulator), and complex strategy (StarCraft Multi-Agent Challenge). They specifically tested these across different dataset qualities to measure robustness to noisy or suboptimal demonstrations.

Key Findings: * Outperforming the State-of-the-Art: The proposed method consistently outperformed baselines like MADT (Multi-Agent Decision Transformer) and MA-CQL across almost all tasks. * Robustness to Poor Data: In the Highway environment with poor-quality data, the authors' method achieved a score of ~17.1, while standard behavior cloning lagged significantly at ~8.7. The teacher acts as a filter, finding moments of accidental brilliance in poor datasets and distilling that refined strategy to the students. * Sample Efficiency and Convergence: The model reached high performance levels with significantly fewer training trajectories than competing offline RL methods. Furthermore, the centralized teacher converged faster than shared-weight models, providing a more stable target for the students.

Limitations and Future Directions

While promising, the framework has limitations. The centralized teacher must process information from all agents simultaneously; as the number of agents grows, the computational cost of the teacher's self-attention mechanism could become a bottleneck. Additionally, like all offline RL, the method is limited by the diversity of the original dataset—if a necessary behavior is completely missing, the teacher cannot invent it.

Future work may explore more advanced distillation techniques and investigate how this structural distillation might apply to heterogeneous agents with completely different roles or abilities.

Summary updated

Abstract

We introduce an offline multi-agent reinforcement learning ( offline MARL) framework that utilizes previously collected data without additional online data collection. Our method reformulates offline MARL as a sequence modeling problem and thus builds on top of the simplicity and scalability of the Transformer architecture. In the fashion of centralized training and decentralized execution, we propose to first train a teacher policy as if the MARL dataset is generated by a single agent. After the teacher policy has identified and recombined the "good" behavior in the dataset, we create separate student policies and distill not only the teacher policy's features but also its structural relations among different agents' features to student policies. Despite its simplicity, the proposed method outperforms state-of-the-art model-free offline MARL baselines while being more robust to demonstration's quality on several environments.
BibTeX
@inproceedings{tseng2022offline,
  author = {Wei-Cheng Tseng and Tsun-Hsuan Johnson Wang and Yen-Chen Lin and Phillip Isola},
  title = {{Offline Multi-Agent Reinforcement Learning with Knowledge Distillation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=5JBJRZtyom},
}
Figure