Abstract
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by compressing models, speeding up learning, and combining multiple tasks into one agent. However, despite its popularity, the RL community has been using a "zoo" of different mathematical formulations for distillation without a clear understanding of how they differ. This paper provides a unified mathematical framework for policy distillation, revealing that many popular "on-policy" methods do not follow a valid gradient. To fix this, the authors introduce expected entropy regularised distillation, a novel method that ensures convergence while significantly outperforming traditional techniques in speed and robustness.
Introduction
In Reinforcement Learning (RL), distillation involves transferring knowledge from a "teacher" policy to a "student" policy. While the high-level objective—making the student's actions mimic the teacher's—is straightforward, implementation details vary significantly. Researchers often choose between sampling data from the teacher (teacher-driven) or the student (student-driven/on-policy).
Empirically, on-policy distillation is superior because it mitigates "distribution shift": the student learns on the states it actually visits. However, this paper identifies a critical theoretical gap: standard on-policy distillation update rules often do not form a valid gradient vector field. Consequently, there is no underlying loss function being minimized, which can lead to training oscillations or failure to converge, especially when environmental rewards are integrated.
The "Non-Gradient" Trap
The authors prove that "On-policy Distill" (sampling from the student) is mathematically problematic. In certain reward-rich environments, this "pseudo-gradient" can cause the policy to loop indefinitely rather than converging to an optimal solution. This finding explains why many practitioners encounter instabilities when combining distillation with standard RL objectives.
Proposed Methods
To resolve these issues, the paper introduces several variants:
- N-distill: A "corrected" version of on-policy distillation that incorporates a reward-based term to restore the gradient property, ensuring mathematical correctness.
- Expected Entropy Regularised Distillation: A novel approach that splits trajectory-level entropy terms into 1-step updates. This maintains mathematical correctness while reducing the high variance typical of trajectory-based methods.
- Teacher V Reward: This method utilizes the teacher’s value function ($V$) as an intrinsic reward (shaping). This allows the student to learn effectively even from imperfect or noisy teachers, potentially surpassing the teacher's performance.
Empirical Results
The authors evaluated these methods across various synthetic environments, including Tabular MDPs, Random Grid Worlds, and Large Action-Space MDPs (up to 4,000 actions).
- Efficiency: Student-driven distillation reached target performance levels 3x faster than teacher-driven methods and 10x faster than uniform sampling.
- Scalability: Expected entropy regularised distillation remained robust in large action spaces where standard methods collapsed due to variance.
- Robustness: The "Teacher V reward" approach allowed students to recover nearly 100% of available rewards even when the teacher was noisy (25% random action rate).
Practical Roadmap
The culmination of this research is a practical decision tree designed to help practitioners select the optimal distillation strategy based on their specific constraints, such as teacher strength and access to value functions.
Conclusion
This work clarifies the theoretical landscape of policy distillation. By identifying the "non-gradient" nature of common on-policy methods and providing robust alternatives like N-distill and expected entropy regularisation, the authors provide a solid foundation for future RL research. While currently tested on synthetic MDPs, these principles offer a clear path toward more stable and efficient distillation in large-scale deep RL applications.
Summary updated