Abstract
Knowledge Distillation (KD) is a critical technique for deploying large-scale models on resource-constrained devices. However, a significant "capacity gap" often exists between massive teacher models and compact student models, where the teacher's highly confident, "sharp" predictions are difficult for the student to replicate. We present PESF-KD (Parameter-Efficient and Student-Friendly Knowledge Distillation), a novel framework that bridges this gap by incorporating small, trainable adapter modules into the teacher. This approach enables the teacher to adapt its knowledge representation to be more "student-friendly" without the prohibitive cost of full-model fine-tuning.
Introduction: The Capacity Gap
In modern deep learning, larger models generally achieve superior performance but are often too computationally expensive for edge deployment. Knowledge Distillation aims to transfer the performance of these large "teachers" to smaller "students." A persistent challenge in KD is the mismatch in model capacity. Large teachers often produce extremely "sharp" predictive distributions—assigning nearly all probability to a single class. Small students lack the parameters to capture these nuances, leading to suboptimal knowledge transfer.
While "online distillation," which involves joint training of the teacher and student, can mitigate this issue, it is computationally intensive as it requires updating billions of parameters. PESF-KD seeks to achieve the benefits of online distillation with a fraction of the computational overhead.
Methodology: PESF-KD
The core of PESF-KD lies in making the teacher model adaptable to the student's learning capacity through two primary mechanisms:
1. Mathematical Sharpness Analysis
The authors identify the capacity mismatch as a "sharpness gap." By theoretically analyzing the variance of the teacher's outputs, they demonstrate that "smoothing" these distributions—without losing inter-class relationships—allows the student to learn more effectively.
2. Parameter-Efficient Adapters
Instead of fine-tuning the entire teacher model, PESF-KD inserts lightweight Adapter modules into the teacher's architecture. These modules follow a bottleneck structure: projecting data to a lower dimension, applying a non-linearity, and projecting it back. During training, the primary teacher parameters remain frozen, and only the adapters are updated based on both ground-truth labels and feedback from the student's performance.
This feedback loop allows the teacher to dynamically "lower its language," adjusting its internal representations to better suit the student's limited capacity.
Experimental Setup
The framework was evaluated across diverse domains:
- Computer Vision: Experiments were conducted on CIFAR-100 using various architectures (ResNet, VGG) and on the large-scale ImageNet dataset.
- Natural Language Understanding (NLU): Using the GLUE benchmark, the researchers tested distillation from a BERT-Base teacher to a 6-layer BERT student across both low-resource (e.g., MRPC, STS-B, RTE) and high-resource (e.g., SST-2, QNLI, QQP) tasks.
Results and Discussion
PESF-KD demonstrates that efficiency does not have to come at the cost of performance.
- Performance vs. Efficiency: On CIFAR-100 (ResNet56-20), PESF-KD achieved 71.63% accuracy, nearly matching the 71.65% of full teacher fine-tuning while reducing training time per batch by over 50% (0.228s vs 0.481s).
- Scalability on ImageNet: On ImageNet, PESF-KD reached 69.96% accuracy. It outperformed leading online distillation methods like KDCL in speed (nearly 3x faster) while significantly narrowing the sharpness gap (reducing it from 31.8 to 18.6).
- Improved Representations: Qualitative analysis indicates that students trained via PESF-KD develop tighter, more distinct feature clusters, suggesting superior internal representation learning compared to vanilla KD.
Conclusion
PESF-KD provides a "student-friendly" interface for large-scale knowledge transfer. By updating only a tiny fraction of the teacher's parameters through adapters, the method achieves state-of-the-art performance with high computational efficiency. Future work may extend this adapter-based approach to more complex tasks such as object detection and text generation, or integrate it into multi-stage distillation pipelines.
Summary updated