In the standard knowledge distillation (KD) paradigm, we assume that a "smarter" teacher—one with more parameters and higher accuracy—will naturally produce a better student. However, research by Jang Hyun Cho and Bharath Hariharan in On the Efficacy of Knowledge Distillation revealed a counterintuitive reality: as the gap between the teacher’s capacity and the student’s capacity grows, the effectiveness of distillation often plateaus or even declines.
This "capacity gap" occurs because a massive teacher model operates in a representational space that a tiny student simply cannot replicate. The teacher’s soft labels become too complex or "too certain" for the student to follow, leading to a mismatched optimization landscape. To solve this, a new direction has emerged: Progressive Distillation, which replaces the single, static teacher with a curriculum of targets that evolve alongside the student.
Structural Stepping Stones: Teacher Assistants
The most direct way to bridge a massive capacity gap is to introduce an intermediate model. Iman Mirzadeh and colleagues introduced this as Teacher Assistant Knowledge Distillation (TAKD). Instead of distilling a 100-layer network directly into a 10-layer network, they first distill the 100-layer model into a 30-layer "assistant." The assistant then serves as the teacher for the final student.
This structural progression works because the assistant shares a more similar architectural "vocabulary" with the student. By breaking one giant leap into two smaller steps, the student is never exposed to a supervision signal that is fundamentally beyond its modeling power. This concept is extended in Partial to Whole Knowledge Distillation (PWKD) by Xiangyu Zhang, where the student first learns from smaller sub-networks (partial knowledge) before tackling the full teacher.
Temporal Curricula: Learning from the Teacher’s History
If structural assistants provide a spatial bridge, Online Distillation provides a temporal one. Instead of using a fully converged, "expert" teacher, researchers have found success in distilling from the teacher’s own training trajectory.
Hrayr Harutyunyan and others argue in Supervision Complexity and its Role in Knowledge Distillation that a student should follow the teacher’s functional evolution. By learning from a sequence of teacher checkpoints—from early, "clumsy" versions to the final expert—the student encounters a natural curriculum.
Recent work by Abhishek Panigrahi and Bingbin Liu in Progressive distillation induces an implicit curriculum provides the intuition for why this is so effective:
- Feature Forgetting: A fully trained teacher often "forgets" or compresses the simple, low-level features (like n-grams in NLP) that it used during early training.
- Signal Alignment: Intermediate checkpoints provide explicit signals for these foundational sub-tasks.
- Sample Complexity: This guidance can reduce the data needed to learn complex tasks from exponential to polynomial, as the student isn't forced to "discover" the basics from an expert who has already moved on to global dependencies.
Adaptive Teaching: The Teacher as a Meta-Learner
Rather than a fixed sequence of models, some researchers propose making the teacher dynamic. In BERT Learns to Teach, Wangchunshu Zhou and Canwen Xu use meta-learning to turn the teacher into an adaptive "professor."
In this framework, the teacher receives feedback on the student’s performance (e.g., on a "quiz set") and adjusts its soft labels accordingly. If the student is struggling, the teacher simplifies the material—perhaps by increasing the "temperature" of its outputs to make the probability distribution smoother and less intimidating. As the student improves, the teacher "sharpens" its targets, gradually increasing the difficulty. This ensures the teacher's guidance remains within the student's "zone of proximal development."
Why Progressive Distillation Works
The unifying intuition across these papers—whether they use intermediate models like PROD by Yeyun Gong or dynamic temperature like CTKD by Zheng Li—is the smoothing of the optimization landscape.
Direct distillation from a high-capacity teacher creates a complex, non-convex objective with many sharp local minima where a small student can get stuck. Progressive distillation transforms this into a sequence of "continuation" problems. By starting with a simplified version of the teacher's knowledge, the student first finds a stable, broad region of the parameter space. As the teacher's complexity increases, the student only needs to perform local refinements rather than global searches.
By managing the supervision complexity, these methods ensure that the student is always learning the most useful features it is currently capable of representing, eventually reaching a performance level that direct distillation from an "over-qualified" teacher could never achieve.