All You Need in Knowledge Distillation Is a Tailored Coordinate System

Junjie Zhou, Ke Zhu, Jianxin WuAAAI 2025 · DOI

TL;DR

This paper introduces a novel knowledge distillation (KD) method called Tailored Coordinate System (TCS). TCS leverages self-supervised learning (SSL) pretrained models as teachers, eliminating the need for task-specific teacher training. By focusing on the coordinate system of teacher features and tailoring it to the student's task, TCS achieves superior accuracy with significantly reduced training time and memory costs compared to existing KD methods.

What Problem Does This Paper Address?

Knowledge Distillation (KD) is a powerful technique for compressing large, complex "teacher" models into smaller, more efficient "student" models while retaining accuracy. However, traditional KD methods suffer from several drawbacks:

  • Reliance on Task-Specific Teachers: They require a teacher model that is specifically trained for the target task, which is costly, time-consuming, and inflexible.
  • Inflexibility: Existing methods often struggle with complex tasks or when there's a large capacity gap between the teacher and student.
  • Inefficiency: Most KD approaches, whether logit-based or feature-based, necessitate multiple forward passes of the teacher model, effectively doubling training time and GPU memory requirements.

The paper aims to address these limitations by proposing a "teacher-free" KD approach that utilizes readily available self-supervised learning (SSL) pretrained models as teachers, making the process more efficient and flexible.

Methods

The core of the proposed method, Tailored Coordinate System (TCS), is built on two key ideas:

  1. Teacher's Coordinate System: The authors hypothesize that the "dark knowledge" within an SSL-pretrained model is encoded in the linear subspace where its features reside. This coordinate system can be efficiently captured using Principal Component Analysis (PCA) on the teacher's features.
  2. Tailoring the Coordinate System: To adapt this general coordinate system to a specific downstream task and the student network, TCS employs an iterative feature selection process. This process prunes irrelevant dimensions from the teacher's feature space, creating a "tailored" coordinate system that is optimal for the student's learning.

The process involves:

  • Feature Extraction: A single forward pass of the teacher model on the training data to extract features.
  • PCA for Coordinate System: Computing PCA on these features to define the teacher's coordinate system.
  • Student Feature Alignment: Aligning the student's features to this tailored coordinate system through projection.
  • Iterative Feature Selection: A trainable mask is used to iteratively select the most relevant dimensions of the projected student features, guided by a loss function that balances the standard cross-entropy loss with an L1-norm penalty on the mask.

For scenarios where a task-specific teacher is available, an "efficient LSH" (eLSH) module is proposed, which mimics features in the teacher's coordinate system without requiring data augmentation, thus maintaining efficiency.

Datasets

The paper evaluates TCS on the following datasets:

  • CIFAR-100: A dataset for image classification.
  • ImageNet-1K: A large-scale dataset for image classification.
  • CUB-200-2011 (CUB): A dataset for bird species recognition, used in few-shot learning experiments.
  • Flowers102: Used for out-of-domain PCA experiments.
  • Pets: Used for out-of-domain PCA experiments.
  • Diabetic Retinopathy: Used for out-of-domain PCA experiments.

The experiments involve both traditional KD settings and practical few-shot learning (pFSL) scenarios.

Key Insights and Ideas

  • Dark Knowledge in Coordinate Systems: The central hypothesis is that a significant portion of a teacher model's dark knowledge is embedded in the linear subspace (coordinate system) of its features, not just the raw feature values or output logits.
  • SSL Pretrained Models as Teachers: The paper demonstrates that SSL-pretrained models, which are readily available and don't require task-specific training, can serve as effective teachers for KD.
  • Tailoring is Key: Simply using a teacher's coordinate system isn't enough; tailoring it through feature selection to the specific target task is crucial for optimal student performance.
  • Efficiency Gains: By requiring only one forward pass of the teacher and integrating the learned coordinate system into the student's classifier, TCS significantly reduces training time and memory usage.
  • Robustness to Capacity Gaps: TCS is shown to be effective even with large differences in model size between the teacher and student, a common challenge in KD.

Key Results

  • Superior Accuracy: TCS consistently achieves higher top-1 accuracy than state-of-the-art KD methods on CIFAR-100 and ImageNet-1K. For example, on ImageNet-1K, TCS achieves up to 81.60% accuracy, outperforming other methods.
  • Reduced Training Time and Memory: TCS requires roughly half the training time and significantly less GPU memory compared to other KD methods. On ImageNet-1K, it uses only 5.6-5.8 GB of GPU memory and takes around 30-32 seconds per epoch, compared to methods using 7.5 GB and taking over 50 seconds.
  • Effective in Few-Shot Learning: In practical few-shot learning scenarios, TCS demonstrates robust performance, often exceeding the IbM2 baseline by 1-2% accuracy, especially as the number of shots increases.
  • Versatility: TCS works well across diverse architectures (CNNs, Transformers, MLPs) and can handle large capacity gaps between teacher and student.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the current evaluation of TCS is primarily for classification tasks.
  • In few-shot learning, TCS's performance can be slightly worse than baselines when the number of training shots is very small (k=1 or 2), as the PCA computation might be based on too few examples.
  • Future Work:
    • Extend TCS to other computer vision tasks like object detection and segmentation.
  • Explore the application of TCS for compressing and accelerating large language models (LLMs) and large multimodal models, leveraging its success with SSL-pretrained teachers.
  • Investigate the theoretical foundations of how dark knowledge is encoded within the coordinate system.

Summary updated

BibTeX
@inproceedings{zhou2025all,
  author = {Junjie Zhou and Ke Zhu and Jianxin Wu},
  title = {{All You Need in Knowledge Distillation Is a Tailored Coordinate System}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34457},
}
Figure