Reinforcement learning (RL) often presents a difficult choice: one can use model-based trajectory optimization to find precise solutions for specific tasks efficiently, or use model-free deep RL to learn general, reactive policies that handle high-dimensional sensory input. The former is sample-efficient but lacks generalization; the latter generalizes well but struggles with the "needle-in-a-haystack" problem of exploration in complex state spaces.
A powerful middle ground has emerged through a process of policy distillation. Instead of training a high-capacity neural network from scratch using sparse environmental rewards, researchers use local trajectory optimizers or symbolic planners as "teachers" to provide dense, high-quality supervision for a "student" policy.
The Teacher-Student Framework
The core intuition behind this direction is that optimization is easier when restricted to a local context. As Levine and Koltun proposed in the foundational Guided Policy Search (GPS) framework, we can use methods like Differential Dynamic Programming to find high-reward trajectories for specific starting conditions. These trajectories serve as a "curriculum," providing a structured roadmap that bypasses the slow trial-and-error process of standard reinforcement learning [Levine and Koltun].
Once these local solutions are found, the problem transforms from reinforcement learning into supervised learning. The global neural network is trained to mimic the action distributions of these local teachers. This decomposition allows the system to bridge the gap between efficient local planning and the need for a versatile, global controller [Levine et al.].
Why Distillation Works: Dense vs. Sparse Signals
Standard RL often fails because the agent receives no feedback until it happens to stumble upon a goal. By using a teacher, the student receives a dense signal at every time step.
- Overcoming Local Optima: By generating "guiding samples" in high-reward regions, trajectory optimization helps complex neural networks avoid poor local optima that trap standard policy gradient methods [Levine and Koltun].
- Sample Efficiency: Levine and Abbeel demonstrated that this approach allows robots to learn complex manipulation skills—like assembling parts or stacking blocks—with only minutes of real-world interaction, as the network learns from low-variance, optimized data rather than noisy exploration [Levine and Abbeel].
- Handling Discontinuities: In tasks involving complex contact dynamics (like opening doors), Chebotar et al. showed that model-free "path integral" teachers can guide a global student through scenarios that would stump traditional smooth-optimization techniques [Chebotar et al.].
Generalization through Aggregation
A single trajectory only covers a narrow slice of the state space. To create a robust policy, the student must distill multiple local controllers—each optimized for different initial conditions—into a single unified model.
Abhishek Gupta showed that this process enables a policy to generalize to unseen states by interpolating between the distinct strategies learned from different demonstrations [Gupta]. Similarly, Chebotar and Hausman demonstrated that distilling simple, time-varying linear-Gaussian controllers into a deep neural network allows these local laws to scale to broader state spaces and varying environments [Chebotar and Hausman].
Beyond Neural Networks: Symbolic Distillation
The distillation principle is not limited to neural-to-neural transfers. It is increasingly used to bridge the gap between "black-box" deep RL and interpretable logic:
- Symbolic Policies: Zhang and Zhou found that training a symbolic mathematical formula to mimic a pre-trained deep neural network is significantly more effective than evolving a symbolic policy from scratch. The teacher's dense behavior provides the necessary guidance to find transparent, high-performing expressions [Zhang and Zhou].
- Planning Guidance: In automated planning, Toyer et al. used traditional symbolic planners as teachers to provide optimal action labels for neural policies, allowing the networks to generalize across different problem sizes within the same domain [Toyer et al.].
- Heuristic Internalization: By training models to predict the actions of high-performance engines (like Stockfish for chess), researchers have shown that neural architectures can internalize complex heuristics without needing explicit search algorithms [Mészáros et al.].
In all these cases, the underlying mechanism remains the same: use a specialized, local, or computationally expensive "expert" to generate a path, then use supervised learning to distill that expertise into a fast, reactive, and generalizable "student."
Key Foundational Work
Guided Policy Search (2013) — contributed 2 concept elements to this direction.
Learning Neural Network Policies with Guided Policy Search under Unknown Dynamics (2014) — contributed 1 concept elements to this direction.
Learning Complex Neural Network Policies with Trajectory Optimization (2014) — contributed 1 concept elements to this direction.