Distilling Policy Distillation

Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, Max Jaderberg

Abstract: The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains. Despite the widespread use and conceptual simplicity of distillation, many different formulations are used in practice, and the subtle variations between them can often drastically change the performance and the resulting objective that is being optimised. In this work, we rigorously explore the entire landscape of policy distillation, comparing the motivations and strengths of each variant through theoretical and empirical analysis. Our results point to three distillation techniques, that are preferred depending on specifics of the task. Specifically a newly proposed expected entropy regularised distillation allows for quicker learning in a wide range of situations, while still guaranteeing convergence.

Introduction

Policy distillation—the process of transferring knowledge from a "teacher" agent to a "student" agent—is a cornerstone of modern Reinforcement Learning (RL). It is used for model compression, multi-task learning, and "kickstarting" new agents. However, the field lacks a unified theoretical framework, leading to a "wild west" of implementations. Researchers at DeepMind have addressed this by providing a rigorous "map" of the mathematical landscape of policy distillation, identifying why common methods fail and proposing robust alternatives.

Theoretical Insights: The Non-Gradient Problem

A significant finding of this research is the "non-gradient" nature of standard on-policy distillation. While sampling from the student's own experience (on-policy) often yields better practical results than sampling from the teacher, the authors prove that standard on-policy distillation does not form a valid gradient vector field.

In certain scenarios, such as the chain-structured MDP shown in Figure 3, this lack of a true gradient causes the student's policy to oscillate indefinitely rather than converging to the teacher's performance. To resolve this, the authors introduce N-distill, which adds a reward-based correction term to ensure the update follows a valid gradient, restoring convergence guarantees.

Comparison of standard on-policy distillation (oscillating) vs. the corrected N-distill method (converging) on a chain MDP.

A Taxonomy of Distillation Methods

The paper categorizes several variants:

  • Teacher Distill: Sampling from the teacher policy.
  • On-policy Distill: Sampling from the student policy.
  • Expected Entropy Regularised Distillation: A novel method combining traditional distillation with entropy regularization, specifically designed for stability in large action spaces.
  • Actor-Critic Variants: Utilizing the teacher’s value function ($V_\pi$) as a "gate" to filter learning or as an intrinsic reward.

The direction of the KL divergence also plays a critical role. Cross-entropy ($H^\times(\pi \| \pi_\theta)$) encourages "mean-seeking" behavior, where the student replicates the teacher's full distribution. Reversing the objective ($H^\times(\pi_\theta \| \pi)$) leads to "mode-seeking" behavior, resulting in sharper, more deterministic policies.

Empirical Evaluation and Results

The authors conducted large-scale experiments across 1,000 unique grid worlds and MDPs with action spaces up to 4,000. Key findings include:

  • Efficiency: Student-driven distillation reached teacher-level performance 3x faster than teacher-driven methods and 10x faster than uniform sampling.
  • Scalability: The proposed "Expected Entropy Regularised" method was uniquely robust as action spaces scaled.
  • Resilience: By using the teacher's value function as a gate, students could outperform noisy teachers by selectively learning from high-value decisions.

The Distillation Roadmap

To assist practitioners, the authors synthesized their findings into a practical decision tree. This guide helps select the optimal strategy based on factors like teacher strength, the need for convergence guarantees, and the priority of improvement speed over stability.

Decision tree for selecting the optimal policy distillation strategy.

Conclusion and Limitations

While this work provides a foundational theoretical map, it was primarily conducted in tabular MDPs. The interaction between these objectives and the high-dimensional function approximation of deep neural networks remains an area for future exploration. Nonetheless, the framework and the proposed expected entropy regularised distillation offer a more principled approach to knowledge transfer in RL.

Summary updated

Abstract

The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the widespread use and conceptual simplicity of distillation, many different formulations are used in practice, and the subtle variations between them can often drastically change the performance and the resulting objective that is being optimised. In this work, we rigorously explore the entire landscape of policy distillation, comparing the motivations and strengths of each variant through theoretical and empirical analysis. Our results point to three distillation techniques, that are preferred depending on specifics of the task. Specifically a newly proposed expected entropy regularised distillation allows for quicker learning in a wide range of situations, while still guaranteeing convergence.
BibTeX
@inproceedings{czarnecki2019distilling,
  author = {Wojciech Marian Czarnecki and Razvan Pascanu and Simon Osindero and Siddhant M. Jayakumar and Grzegorz Swirszcz and Max Jaderberg},
  title = {{Distilling Policy Distillation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2019},
  url = {https://openreview.net/forum?id=41cgA4kejeH},
}
Figure