Dropout Connects Transformers and CNNs: Transfer General Knowledge for Knowledge Distillation

Bokyeung Lee, Jonghwan Hong, Hyunuk Shin, Bonwha Ku, Hanseok KoWACV 2025

TL;DR

This paper introduces DropKD, a novel knowledge distillation framework designed to effectively transfer knowledge from powerful Transformer models to more efficient CNN models. By intelligently separating "general" and "specific" knowledge and using dropout to filter out the latter, DropKD significantly improves CNN performance in Transformer-to-CNN distillation tasks, outperforming existing methods.

What Problem Does This Paper Address?

Transformers have revolutionized fields like computer vision and audio processing due to their ability to capture long-range dependencies. However, their high computational cost makes them impractical for deployment on edge devices. Convolutional Neural Networks (CNNs), on the other hand, are computationally efficient and possess strong inductive biases, making them ideal for such scenarios. Knowledge Distillation (KD) offers a way to leverage the performance of Transformers by training smaller CNNs to mimic them.

The core challenge in Transformer-to-CNN KD is the "capacity gap" arising from fundamental structural differences between the two architectures. Existing KD methods often struggle because they try to transfer all of the teacher's knowledge, including "specific" knowledge that is tied to the Transformer's unique architecture and is difficult for a CNN to learn. This leads to performance degradation and overfitting in the student CNN. This paper aims to bridge this gap by selectively transferring only the "general" knowledge that is transferable and beneficial for the CNN student.

Methods

The proposed DropKD framework tackles the capacity gap by introducing a novel approach to knowledge distillation that focuses on separating and transferring "general" knowledge while discarding "specific" knowledge.

The key innovations are:

  • General vs. Specific Knowledge: The authors define "general knowledge" as transferable feature representations useful for the task, and "specific knowledge" as architecture-correlated representations unique to the teacher (e.g., due to its self-attention mechanism).
  • Feature Dropout Distillation: This method uses encoder-decoder architectures for both teacher and student. Dropout layers are strategically placed within the encoders. This dropout mechanism forces the encoders to learn sparse representations, effectively filtering out specific knowledge and extracting general feature representations. The goal is to minimize the disparity between the general knowledge representations of the teacher and student.
  • Logit Dropout Distillation: This component applies dropout to the classifiers of both the teacher and student. This introduces uncertainty in the output logits, allowing the student to learn general knowledge from the teacher's softened predictions while retaining its own specific knowledge, preventing it from simply mimicking the teacher.
  • Sparse PCA: While not the core distillation mechanism, sparse PCA is mentioned as a tool to enhance interpretability and identify principal components that represent general knowledge.

The overall loss function combines cross-entropy loss, logit dropout distillation loss, reconstruction loss, and feature dropout distillation loss, with hyperparameter tuning to balance these components.

Datasets

The paper evaluates DropKD on several challenging image classification datasets:

  • RAF-DB: A dataset for facial expression recognition, featuring diverse facial images.
  • CIFAR-100: A standard image classification dataset.
  • ImageNet-100: A subset of ImageNet, also used for image classification.

These datasets are chosen to represent various scenarios, including those with small classes and those requiring the model to capture detailed image components.

Key Insights and Ideas

  • The "Specific Knowledge" Bottleneck: The paper highlights that the primary reason for poor performance in Transformer-to-CNN KD is the attempt to transfer architecture-specific knowledge, which CNNs cannot effectively learn.
  • Dropout as a Knowledge Filter: Dropout, traditionally used for regularization, is repurposed here as a powerful tool to actively filter out specific knowledge, enabling the transfer of more generalizable representations.
  • General Knowledge is Key: The core idea is that focusing on transferring general knowledge, rather than all knowledge, is crucial for successful distillation between vastly different architectures like Transformers and CNNs.
  • CKA and Dropout are Inversely Related: The study finds that higher Centered Kernel Alignment (CKA) similarity between teacher and student features doesn't necessarily lead to better performance. In fact, DropKD often achieves lower CKA similarity but superior accuracy, suggesting that a strong focus on specific knowledge (which increases CKA) can be detrimental. Optimal dropout rates are found to be inversely proportional to CKA similarity.

Key Results

DropKD demonstrates significant improvements over existing state-of-the-art KD methods across various Transformer-to-CNN scenarios.

  • Superior Accuracy: On RAF-DB, DropKD achieved up to a 4.69% improvement over the baseline. On ImageNet-100, it outperformed other methods by over 2% in ShuffleNetV1 and ShuffleNetV2. On CIFAR-100, DropKD also achieved the best top-1 and top-5 accuracies.
  • Robustness: The method shows robustness across different teacher-student architecture pairings and datasets, indicating its general applicability.
  • Optimal Hyperparameters: Ablation studies identified optimal dropout rates (e.g., 0.5 for encoders and classifiers) and regularization parameters that maximize performance.

Limitations and Future Work

  • Focus on Latent Vectors and Logits: The current DropKD method primarily focuses on distilling knowledge from latent vectors and logits. The authors suggest that while effective for computer vision tasks, its direct application to low-level vision tasks like image super-resolution or denoising might require adaptation, though the core concept could be extended.
  • Further Exploration of CKA: The paper verifies the inverse relationship between CKA similarity and optimal dropout rate, providing a guideline for designing dropout strategies. Future work could further explore this relationship and its implications for KD.

Summary updated

BibTeX
@inproceedings{lee2025dropout,
  author = {Bokyeung Lee and Jonghwan Hong and Hyunuk Shin and Bonwha Ku and Hanseok Ko},
  title = {{Dropout Connects Transformers and CNNs: Transfer General Knowledge for Knowledge Distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure