Abstract
Knowledge Distillation (KD) typically follows a teacher-centered paradigm where a pre-trained teacher model transfers its entire knowledge base to a student. This paper proposes Student-Centered Distillation (SCD), which utilizes a validation set feedback loop and a Fuzzy PID controller to dynamically adjust the teacher's instruction based on the student's specific learning needs. By addressing knowledge deficiency, hard sample learning, and forgetting, SCD ensures the teacher provides knowledge that the student can effectively absorb. Experimental results across multiple benchmarks demonstrate that SCD consistently outperforms state-of-the-art KD methods.
Introduction
In traditional Knowledge Distillation, a large, powerful "teacher" model guides a smaller "student" model by encouraging it to mimic the teacher's outputs. However, most existing methods are teacher-centered: the teacher is pre-trained on its own terms and then "crams" its entire knowledge base into the student. This approach is often inefficient because the teacher and student have different architectures; the teacher might prioritize information that the student cannot absorb or doesn't need. This leads to three major hurdles: knowledge deficiency, hard sample learning, and knowledge forgetting.
Student-Centered Distillation (SCD)
The authors introduce Student-Centered Distillation (SCD), a framework that shifts the focus to what a student "needs to learn." Inspired by human pedagogy, SCD treats the validation set as an "examination" to diagnose the student's weaknesses.
The process involves a two-step online distillation. In each iteration, the teacher first learns the student's needs on the validation set. Then, it transfers that specific, "desired" knowledge to the student using the training set. This feedback loop ensures the teacher's instruction is always aligned with the student's current learning state.
The CL-FPID Algorithm
The core of SCD is the Curriculum Learning-based Fuzzy PID (CL-FPID) algorithm. Borrowed from automation engineering, the PID controller weighs the importance of training samples based on the student's performance:
- Proportional (P): Focuses on current error (what the student doesn't know now).
- Integral (I): Tracks cumulative error to identify "hard" samples that the student consistently fails to master.
- Derivative (D): Monitors error trends to detect and prevent "knowledge forgetting."
Instead of static weights, a fuzzy logic strategy is used. As the student's performance improves through a "curriculum," the system dynamically reduces the focus on simple errors (P) and increases the focus on hard samples (I) and forgetting prevention (D).
Key Results and Insights
SCD was evaluated across various tasks including image classification, object detection, and face verification. It consistently outperformed over 20 state-of-the-art KD methods. For instance, on CIFAR-100 (ResNet-56 to ResNet-20), SCD achieved 71.41% accuracy, surpassing the previous SOTA, MetaDistil (71.25%).
Key insights include:
- Feedback Loops: Using the validation set allows the student to "inform" the teacher about struggles without direct parameter updates on that data.
- Alignment over Accuracy: A teacher with higher raw accuracy isn't always better; effectiveness depends on the alignment between instruction and student capacity.
- Efficiency: As an online method, SCD eliminates the need for expensive teacher pre-training, lowering overall training costs.
Conclusion
SCD represents a creative bridge between control theory and deep learning. By prioritizing the student's learning state through a PID-controlled feedback loop, it offers a more efficient and effective path for knowledge transfer in neural networks. Future work may explore more complex student feedback beyond simple loss values, potentially incorporating structural or feature-map-level feedback into the PID loop.
Summary updated