Interactive Distillation for Cooperative Multi-Agent Reinforcement Learning

Minwoo Cho, Batuhan Altundas, Matthew GombolayarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces HINT, a novel framework for cooperative multi-agent reinforcement learning (MARL) that uses interactive knowledge distillation to train decentralized agents more effectively. HINT addresses key challenges in MARL, such as synthesizing high-quality teacher policies and handling out-of-distribution states, by employing a hierarchical teacher and a unique pseudo off-policy reinforcement learning approach, leading to significant performance improvements.

What Problem Does This Paper Address?

Multi-agent reinforcement learning (MARL) is crucial for tasks requiring coordinated action among multiple agents, like search and rescue or tactical combat. However, training these systems is notoriously difficult due to challenges like partial observability, non-stationary dynamics, and the credit assignment problem. A popular paradigm, Centralized Training with Decentralized Execution (CTDE), uses global information during training but restricts agents to local observations during execution.

Knowledge Distillation (KD) offers a way to accelerate MARL by having a powerful, centralized "teacher" agent guide simpler "student" agents. However, traditional KD faces several bottlenecks:

  1. Teacher Synthesis: Creating a high-performing teacher policy, especially in complex, dynamic environments, is challenging.
  2. Out-of-Distribution (OOD) States: Student agents can encounter states that the teacher hasn't seen during training, leading to unreliable guidance.
  3. Observation Mismatches: The teacher's global view might differ significantly from a student's local view, making it hard for the teacher to provide relevant feedback.

This paper aims to overcome these limitations by proposing HINT, a framework that makes KD more robust and effective for cooperative MARL.

Methods

HINT is built upon three core components:

  • Hierarchical Teacher (HiT-MAC): Instead of a flat teacher policy, HINT uses a hierarchical reinforcement learning (RL) approach. This decomposes decisions into high-level coordination (e.g., assigning subgoals) and low-level execution (e.g., primitive actions). This structure improves scalability and makes the teacher more effective.
  • Pseudo Off-Policy RL: This is a key innovation. HINT updates the teacher policy not only with its own experience but also with the student's trajectories. This allows the teacher to learn how to recover from student mistakes and adapt its guidance to the student's current behavior, especially in OOD states. It leverages techniques from off-policy correction methods like V-trace to stabilize training.
  • Performance-Based Filtering (DAgger with Filtering): To ensure the quality of the guidance provided by the teacher, HINT incorporates a filtering mechanism. When the student queries the teacher for guidance, HINT simulates the teacher's suggested actions to the end of the episode. If the outcome is successful, the guidance is accepted; otherwise, it's rejected. This ensures that only outcome-relevant and high-quality demonstrations are used to train the student, reducing observation mismatches.

Datasets

The authors evaluate HINT on two challenging cooperative MARL environments:

  • MARINE: A maritime logistics environment involving routing and refueling agents navigating dynamic ocean conditions with weather-induced uncertainty.
  • FireCommander (FC): A grid-based wildfire environment where agents must coordinate to extinguish fires, modeled after the FARSITE simulator, incorporating dynamic fire propagation and differential equations.

Both environments are tested across three difficulty levels (easy, medium, hard) to assess scalability.

Key Insights and Ideas

  • Hierarchical Teachers are Better Teachers: Using a hierarchical RL structure for the teacher agent significantly improves its ability to manage complex coordination tasks and provide more effective guidance.
  • Teacher Adaptation is Crucial: The teacher shouldn't be static. Allowing the teacher to learn from student experiences (pseudo off-policy RL) and filtering its guidance based on outcomes (performance-based filtering) is essential for robust performance, especially when students encounter novel situations.
  • Bridging the Teacher-Student Gap: The core challenge in KD for MARL is the distribution shift between teacher and student. HINT's pseudo off-policy RL and performance-based filtering directly address this by making the teacher's guidance more adaptive and relevant to the student's current state and capabilities.
  • OOD States are a Major Bottleneck: The paper empirically demonstrates (Fig. 3) that as task complexity increases, student state distributions diverge significantly from the teacher's, leading to OOD states and reduced teacher effectiveness. HINT's methods are specifically designed to mitigate this.

Key Results

HINT consistently outperforms competitive CTDE and KD baselines across both MARINE and FireCommander environments.

  • Significant Performance Gains: HINT achieved improvements of 60% to 165% in success rate compared to baselines.
  • Robustness in Hard Scenarios: In challenging "Hard" settings (e.g., MARINE-Hard, FC-Hard), HINT showed particularly strong performance, outperforming warm-started baselines and demonstrating a more stable training process with lower standard deviation.
  • Efficiency: While HINT incorporates more complex modules, its overall training time was comparable to or better than some baselines, especially when considering the performance gains. For instance, HINT achieved faster training than HAPPO and HetNet on MARINE-Hard while delivering superior results.

Limitations and Future Work

  • Teacher Scalability: The current teacher policy (HiT-MAC) uses a centralized training paradigm, which inherently has scalability limitations. While hierarchical RL helps, it still scales to a maximum of 10 agents in the tested environments.
  • Human Expertise for Hierarchy: HINT relies on human expertise to define the hierarchical structures. Future work could explore automating this hierarchy construction through skill discovery and macro-action learning.
  • Computational Cost of Filtering: The performance-based filtering mechanism, while effective, can be computationally expensive. Future research could focus on developing more efficient inference modules to evaluate teacher demonstrations.
  • End-to-End Learning: The current approach uses pre-trained low-level policies. Exploring fully end-to-end learning for hierarchical structures could lead to more adaptive behaviors and emergent capabilities.

Summary updated

Abstract

Knowledge distillation (KD) has the potential to accelerate MARL by employing a centralized teacher for decentralized students but faces key bottlenecks. Specifically, there are (1) challenges in synthesizing high-performing teaching policies in complex domains, (2) difficulties when teachers must reason in out-of-distribution (OOD) states, and (3) mismatches between the decentralized students' and the centralized teacher's observation spaces. To address these limitations, we propose HINT (Hierarchical INteractive Teacher-based transfer), a novel KD framework for MARL in a centralized training, decentralized execution setup. By leveraging hierarchical RL, HINT provides a scalable, high-performing teacher. Our key innovation, pseudo off-policy RL, enables the teacher policy to be updated using both teacher and student experience, thereby improving OOD adaptation. HINT also applies performance-based filtering to retain only outcome-relevant guidance, reducing observation mismatches. We evaluate HINT on challenging cooperative domains (e.g., FireCommander for resource allocation, MARINE for tactical combat). Across these benchmarks, HINT outperforms baselines, achieving improvements of 60% to 165% in success rate.
BibTeX
@misc{cho2026interactive,
  author = {Minwoo Cho and Batuhan Altundas and Matthew Gombolay},
  title = {{Interactive Distillation for Cooperative Multi-Agent Reinforcement Learning}},
  year = {2026},
  eprint = {2601.05407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.05407},
}
Figure