Feature Kernel Distillation

Bobby He, Mete OzayICLR 2022 · OpenReview

Introduction

Traditional Knowledge Distillation (KD) typically works by matching the "soft labels" (output probabilities) of a teacher model to a student model. However, this approach has a major limitation: the teacher and student must share the exact same prediction space. If you have a powerful teacher trained on 1,000 classes and want to train a student for a specific 10-class task, vanilla KD cannot be easily applied.

Furthermore, while KD is empirically successful, the underlying reasons for its efficacy remain somewhat mysterious. Existing "feature-based" distillation methods—which look at internal network representations—are often heuristically motivated. To address these gaps, researchers have introduced Feature Kernel Distillation (FKD), providing both a principled theoretical framework based on feature learning and a practical algorithm that works across different architectures and datasets.

Methods: Feature Kernel Distillation

The core idea behind FKD is to view the last layer of a trained neural network as a data-dependent kernel machine. Instead of matching output probabilities, FKD distills knowledge by matching the feature kernels $k(x, x')$—the inner product of representations for two different inputs—between the teacher and the student.

Feature Kernel Distillation architecture

Key Innovations

  1. The Correlation Kernel: Instead of matching raw feature kernels, FKD uses a normalized version called the correlation kernel. This ensures the model focuses on the relationships between different inputs rather than the scale of individual features. Crucially, this "zeros out" diagonal differences, which mathematically prevents the student from simply memorizing noise.
  2. Feature Regularization (FR): To prevent the student's features from collapsing or becoming degenerate during training, the authors introduce an additive $\ell_2$ penalty on the feature norms.
  3. Theoretical Grounding: The authors extend a "multi-view" theoretical framework, proving that different initializations of a network learn different subsets of data attributes (e.g., "wheels" vs. "headlights" for a car). FKD allows a student to "collect" and learn the union of all views captured by an ensemble of teachers.

Key Insights

  • NNs as Kernel Machines: A trained neural network essentially performs kernel ridge regression where the "kernel" is learned from the data. Distillation is the process of transferring this learned similarity metric.
  • The Power of Ensembles: Theory shows that ensembling works because different models "win the lottery" for different data attributes. FKD is uniquely suited for ensemble distillation because it can aggregate these disparate kernels into a single, robust target for the student.
  • Task Agnosticism: Because FKD operates on the feature layer (before the final classification head), it is "prediction-space independent." You can distill knowledge from a teacher trained on one dataset into a student trained on a completely different one.

Key Results

The paper evaluates FKD across a wide range of standard computer vision benchmarks, including CIFAR-10/100, ImageNet-1K, STL-10, and Tiny-ImageNet.

  • Outperforms Baselines: FKD consistently beats vanilla KD and other feature-based methods like Relational KD (RKD) and Similarity-Preserving (SP) distillation across various architectures.
  • Dataset Transfer: Because FKD does not rely on matching class logits, it excels at cross-dataset transfer. In a CIFAR-100 $\rightarrow$ STL-10 transfer task, FKD improved student accuracy by 2.75% over standard training, whereas vanilla KD could not be used at all.
Dataset transfer performance comparing FKD to baselines
  • Ensemble Efficiency: A single student distilled from an ensemble of two teachers using FKD can match the performance of the teacher ensemble itself.
  • State-of-the-Art: On CIFAR-100, FKD matches or exceeds the performance of Contrastive Representational Distillation (CRD), a significantly more computationally expensive method.

Limitations and Future Work

While highly effective, FKD does have limitations. Calculating pairwise similarities within a minibatch adds a computational cost that scales with the square of the batch size. Additionally, the "multi-view" data setup used for the theoretical proofs is a simplified proxy for real-world data, and the method requires tuning hyperparameters for distillation strength and feature regularization.

Future work may involve applying FKD to more complex self-supervised learning settings or exploring how to distill knowledge from the intermediate layers of the network using similar kernel-based principles.

Summary updated

Abstract

Trained Neural Networks (NNs) can be viewed as data-dependent kernel machines, with predictions determined by the inner product of last-layer representations across inputs, referred to as the feature kernel. We explore the relevance of the feature kernel for Knowledge Distillation (KD), using a mechanistic understanding of an NN’s optimisation process. We extend the theoretical analysis of Allen-Zhu & Li (2020) to show that a trained NN’s feature kernel is highly dependent on its parameter initialisation, which biases different initialisations of the same architecture to learn different data attributes in a multi-view data setting. This enables us to prove that KD using only pairwise feature kernel comparisons can improve NN test accuracy in such settings, with both single & ensemble teacher models, whereas standard training without KD fails to generalise. We further use our theory to motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to propose a novel approach: Feature Kernel Distillation (FKD). Finally, we experimentally corroborate our theory in the image classification setting, showing that FKD is amenable to ensemble distillation, can transfer knowledge across datasets, and outperforms both vanilla KD & other feature kernel based KD baselines across a range of standard architectures & datasets.
BibTeX
@inproceedings{he2022feature,
  author = {Bobby He and Mete Ozay},
  title = {{Feature Kernel Distillation}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=tBIQEvApZK5},
}
Figure