Introduction
Traditional Knowledge Distillation (KD) typically works by matching the "soft labels" (output probabilities) of a teacher model to a student model. However, this approach has a major limitation: the teacher and student must share the exact same prediction space. If you have a powerful teacher trained on 1,000 classes and want to train a student for a specific 10-class task, vanilla KD cannot be easily applied.
Furthermore, while KD is empirically successful, the underlying reasons for its efficacy remain somewhat mysterious. Existing "feature-based" distillation methods—which look at internal network representations—are often heuristically motivated. To address these gaps, researchers have introduced Feature Kernel Distillation (FKD), providing both a principled theoretical framework based on feature learning and a practical algorithm that works across different architectures and datasets.
Methods: Feature Kernel Distillation
The core idea behind FKD is to view the last layer of a trained neural network as a data-dependent kernel machine. Instead of matching output probabilities, FKD distills knowledge by matching the feature kernels $k(x, x')$—the inner product of representations for two different inputs—between the teacher and the student.
Key Innovations
- The Correlation Kernel: Instead of matching raw feature kernels, FKD uses a normalized version called the correlation kernel. This ensures the model focuses on the relationships between different inputs rather than the scale of individual features. Crucially, this "zeros out" diagonal differences, which mathematically prevents the student from simply memorizing noise.
- Feature Regularization (FR): To prevent the student's features from collapsing or becoming degenerate during training, the authors introduce an additive $\ell_2$ penalty on the feature norms.
- Theoretical Grounding: The authors extend a "multi-view" theoretical framework, proving that different initializations of a network learn different subsets of data attributes (e.g., "wheels" vs. "headlights" for a car). FKD allows a student to "collect" and learn the union of all views captured by an ensemble of teachers.
Key Insights
- NNs as Kernel Machines: A trained neural network essentially performs kernel ridge regression where the "kernel" is learned from the data. Distillation is the process of transferring this learned similarity metric.
- The Power of Ensembles: Theory shows that ensembling works because different models "win the lottery" for different data attributes. FKD is uniquely suited for ensemble distillation because it can aggregate these disparate kernels into a single, robust target for the student.
- Task Agnosticism: Because FKD operates on the feature layer (before the final classification head), it is "prediction-space independent." You can distill knowledge from a teacher trained on one dataset into a student trained on a completely different one.
Key Results
The paper evaluates FKD across a wide range of standard computer vision benchmarks, including CIFAR-10/100, ImageNet-1K, STL-10, and Tiny-ImageNet.
- Outperforms Baselines: FKD consistently beats vanilla KD and other feature-based methods like Relational KD (RKD) and Similarity-Preserving (SP) distillation across various architectures.
- Dataset Transfer: Because FKD does not rely on matching class logits, it excels at cross-dataset transfer. In a CIFAR-100 $\rightarrow$ STL-10 transfer task, FKD improved student accuracy by 2.75% over standard training, whereas vanilla KD could not be used at all.
- Ensemble Efficiency: A single student distilled from an ensemble of two teachers using FKD can match the performance of the teacher ensemble itself.
- State-of-the-Art: On CIFAR-100, FKD matches or exceeds the performance of Contrastive Representational Distillation (CRD), a significantly more computationally expensive method.
Limitations and Future Work
While highly effective, FKD does have limitations. Calculating pairwise similarities within a minibatch adds a computational cost that scales with the square of the batch size. Additionally, the "multi-view" data setup used for the theoretical proofs is a simplified proxy for real-world data, and the method requires tuning hyperparameters for distillation strength and feature regularization.
Future work may involve applying FKD to more complex self-supervised learning settings or exploring how to distill knowledge from the intermediate layers of the network using similar kernel-based principles.
Summary updated