Cumulative Spatial Knowledge Distillation for Vision Transformers

Borui Zhao, Renjie Song, Jiajun Liang

Abstract

Knowledge distillation from Convolutional Neural Networks (CNNs) to Vision Transformers (ViTs) has become a standard technique to introduce local inductive biases into data-hungry Transformer architectures. However, this process often creates a "capacity ceiling" where the student is forced to mimic the teacher's local limitations, suppressing the ViT's inherent global modeling potential. We present Cumulative Spatial Knowledge Distillation (CSKD), a framework that simplifies distillation by bypassing intermediate feature alignment and employing a dynamic fusion strategy. By shifting supervision from local spatial responses to global information as training progresses, CSKD allows ViTs to leverage CNN-like efficiency early on without sacrificing long-range dependency modeling.

Introduction

Training Vision Transformers from scratch is challenging because they lack the "image-friendly" assumptions, such as translation invariance, built into CNNs. While Knowledge Distillation (KD) using a pre-trained CNN teacher can guide a ViT student, it faces two major hurdles. First, the architectural mismatch between CNNs and ViTs makes aligning intermediate features inefficient, often requiring complex auxiliary modules. Second, CNNs are inherently limited to local patterns; forcing a ViT to follow a CNN teacher too closely throughout training can prevent the student from reaching its full global potential.

Methodology

CSKD addresses these challenges through two primary components: Spatial-wise Knowledge Transfer and Cumulative Knowledge Fusion (CKF).

Spatial-wise Knowledge Transfer

Instead of matching intermediate layers, CSKD focuses on the final output. The CNN's last feature map is treated as a set of independent local classifiers. These "dense predictions" supervise the corresponding patch tokens in the ViT, transferring spatial knowledge directly without requiring architectural re-engineering of the student.

Cumulative Knowledge Fusion (CKF)

The CKF module introduces a dynamic fusion strategy that balances local and global supervision. In the early stages of training, the fusion weight $\alpha$ is high, prioritizing local spatial responses to help the ViT learn basic image structures. As training progresses, $\alpha$ decays, shifting the focus toward global responses. This "graduation" effect allows the ViT to eventually focus on the global relations it was designed to model.

The CSKD framework, illustrating the CNN teacher, ViT student, and the Cumulative Knowledge Fusion module that dynamically adjusts supervision.

The total loss is defined as a combination of standard cross-entropy ($\mathcal{L}_{CE}$), the original DeiT distillation loss ($\mathcal{L}_{distill}$), and the proposed CSKD loss: $$\mathcal{L} = \frac{1}{2}\mathcal{L}_{CE} + \frac{1}{2}\mathcal{L}_{distill} + \mathcal{L}_{CSKD}$$

Experimental Results

The researchers evaluated CSKD on ImageNet-1k and several transfer learning tasks. CSKD-Ti (Tiny) achieved 76.3% top-1 accuracy on ImageNet-1k, a +1.8% improvement over the DeiT baseline. When extended to 1000 epochs, CSKD-Ti reached 78.1%, outperforming other specialized distillation methods like DearKD.

Global Modeling Capability

A key finding of this work is the impact on the ViT's receptive field. By visualizing the "mean attention distance," the authors demonstrated that CSKD models exhibit significantly higher attention distances in later layers compared to standard DeiT. This empirical evidence confirms that the cumulative strategy successfully encourages the ViT to integrate information across the entire image rather than remaining tethered to local patches.

Comparison of mean attention distance across network depth between standard DeiT and the proposed distillation method.

Qualitative heatmaps further support these findings, showing that CSKD-trained models focus more accurately on salient objects compared to baselines. The method also showed strong generalization on transfer learning tasks, notably achieving a +1.1% improvement on the iNaturalist 2019 dataset.

Conclusion

Cumulative Spatial Knowledge Distillation provides an elegant solution to the trade-off between local inductive bias and global modeling capacity. By simplifying the distillation pipeline and introducing a dynamic "graduation" mechanism, CSKD enables Vision Transformers to surpass the limitations of their CNN teachers. While the method introduces some training overhead, its performance gains and improved global attention make it a robust choice for efficient ViT training.

Summary updated

Abstract

Distilling knowledge from convolutional neural networks (CNNs) is a double-edged sword for vision transformers (ViTs). It boosts the performance since the image-friendly local-inductive bias of CNN helps ViT learn faster and better, but leading to two problems: (1) Network designs of CNN and ViT are completely different, which leads to different semantic levels of intermediate features, making spatial-wise knowledge transfer methods (e.g., feature mimicking) inefficient. (2) Distilling knowledge from CNN limits the network convergence in the later training period since ViT's capability of integrating global information is suppressed by CNN's local-inductive-bias supervision. To this end, we present Cumulative Spatial Knowledge Distillation (CSKD). CSKD distills spatial-wise knowledge to all patch tokens of ViT from the corresponding spatial responses of CNN, without introducing intermediate features. Furthermore, CSKD exploits a Cumulative Knowledge Fusion (CKF) module, which introduces the global response of CNN and increasingly emphasizes its importance during the training. Applying CKF leverages CNN's local inductive bias in the early training period and gives full play to ViT's global capability in the later one. Extensive experiments and analysis on ImageNet-1k and downstream datasets demonstrate the superiority of our CSKD. Code will be publicly available.
BibTeX
@inproceedings{zhao2023cumulative,
  author = {Borui Zhao and Renjie Song and Jiajun Liang},
  title = {{Cumulative Spatial Knowledge Distillation for Vision Transformers}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00565},
}
Figure