Abstract
Knowledge Distillation (KD) aims to transfer knowledge from a complex teacher model to a compact student model. However, the capacity gap between models often limits performance. We propose IMF (Integrating Matched Features), a framework that utilizes intermediate branch networks and an attentive logit mechanism. By integrating features from multiple stages and employing an "early exit" strategy, IMF allows student models to achieve state-of-the-art accuracy, sometimes surpassing their teachers, while maintaining high parameter efficiency.
Introduction
The primary challenge in Knowledge Distillation is the structural discrepancy between large teacher models and smaller student models. Standard KD approaches typically treat the student architecture as fixed, focusing on matching final logits or intermediate feature maps. This "fixed architecture" constraint creates a bottleneck, as small models may lack the capacity to fully represent the teacher's complex knowledge. IMF addresses this by augmenting the student's internal structure during training with flexible branch networks.
Methodology
The IMF framework is built upon three primary components designed to bridge the capacity gap:
Intermediate Feature Distillers (IFD)
Rather than relying solely on the final layer, IMF attaches "branch networks" called IFDs to the student's intermediate layers. These branches are trained to predict the teacher's output distribution using early-stage features.
Element-wise Normalized Attentive (ENA) Layer
Each IFD branch concludes with an ENA layer. This component acts as a learnable attention mechanism that applies weights to class logits. This allows the model to dynamically prioritize branches that demonstrate higher confidence for specific classes, leading to a more robust integrated prediction.
Integrated Distillation Loss
The training objective combines three loss terms: a standard Cross-Entropy loss ($L_{CE}$), an intermediate feature matching loss ($L_{IF}$), and a novel Integrated Distillation loss ($L_{ID}$) that ensures the combined output of all IFD branches matches the teacher's distribution.
A key advantage of this approach is the "flexible inference" capability. During deployment, the original final layers of the student can be discarded in favor of the integrated IFD outputs, often resulting in a "shorter" and more efficient model than the original student backbone.
Experimental Results
IMF was evaluated on CIFAR-10, CIFAR-100, ImageNet, and the WFLW facial landmark dataset.
Performance and Efficiency
IMF consistently outperforms existing SOTA distillation methods such as CRD, RKD, and SR. Notably, in several configurations, the student model trained via IMF achieves higher accuracy than the teacher model itself. For instance, a VGG8 student distilled from a VGG19 teacher surpassed the teacher's performance.
On ImageNet, IMF improved a ResNet18 student's Top-1 accuracy by 2.63% over the baseline. Furthermore, the method demonstrates significant parameter efficiency; for a ResNet32x4 to ResNet8x4 pair, IMF achieved a 4.91% improvement over the baseline while utilizing only 15.5% of the teacher's parameters.
Robustness
Unlike many KD methods that struggle when the teacher is significantly more complex than the student, IMF maintains stable performance gains across a wide variety of teacher-student architectures, effectively mitigating the "teacher-student mismatch" problem.
Conclusion
IMF introduces a flexible and efficient paradigm for knowledge distillation. By treating intermediate branches as primary predictors and integrating them via an attentive mechanism, it effectively overcomes the capacity gap. Future work will focus on automating branch placement and optimizing hyperparameter selection to further enhance the framework's versatility.
Summary updated