Abstract
Knowledge distillation from Convolutional Neural Networks (CNNs) to Vision Transformers (ViTs) has become a standard technique to introduce local inductive biases into data-hungry Transformer architectures. However, this process often creates a "capacity ceiling" where the student is forced to mimic the teacher's local limitations, suppressing the ViT's inherent global modeling potential. We present Cumulative Spatial Knowledge Distillation (CSKD), a framework that simplifies distillation by bypassing intermediate feature alignment and employing a dynamic fusion strategy. By shifting supervision from local spatial responses to global information as training progresses, CSKD allows ViTs to leverage CNN-like efficiency early on without sacrificing long-range dependency modeling.
Introduction
Training Vision Transformers from scratch is challenging because they lack the "image-friendly" assumptions, such as translation invariance, built into CNNs. While Knowledge Distillation (KD) using a pre-trained CNN teacher can guide a ViT student, it faces two major hurdles. First, the architectural mismatch between CNNs and ViTs makes aligning intermediate features inefficient, often requiring complex auxiliary modules. Second, CNNs are inherently limited to local patterns; forcing a ViT to follow a CNN teacher too closely throughout training can prevent the student from reaching its full global potential.
Methodology
CSKD addresses these challenges through two primary components: Spatial-wise Knowledge Transfer and Cumulative Knowledge Fusion (CKF).
Spatial-wise Knowledge Transfer
Instead of matching intermediate layers, CSKD focuses on the final output. The CNN's last feature map is treated as a set of independent local classifiers. These "dense predictions" supervise the corresponding patch tokens in the ViT, transferring spatial knowledge directly without requiring architectural re-engineering of the student.
Cumulative Knowledge Fusion (CKF)
The CKF module introduces a dynamic fusion strategy that balances local and global supervision. In the early stages of training, the fusion weight $\alpha$ is high, prioritizing local spatial responses to help the ViT learn basic image structures. As training progresses, $\alpha$ decays, shifting the focus toward global responses. This "graduation" effect allows the ViT to eventually focus on the global relations it was designed to model.
The total loss is defined as a combination of standard cross-entropy ($\mathcal{L}_{CE}$), the original DeiT distillation loss ($\mathcal{L}_{distill}$), and the proposed CSKD loss: $$\mathcal{L} = \frac{1}{2}\mathcal{L}_{CE} + \frac{1}{2}\mathcal{L}_{distill} + \mathcal{L}_{CSKD}$$
Experimental Results
The researchers evaluated CSKD on ImageNet-1k and several transfer learning tasks. CSKD-Ti (Tiny) achieved 76.3% top-1 accuracy on ImageNet-1k, a +1.8% improvement over the DeiT baseline. When extended to 1000 epochs, CSKD-Ti reached 78.1%, outperforming other specialized distillation methods like DearKD.
Global Modeling Capability
A key finding of this work is the impact on the ViT's receptive field. By visualizing the "mean attention distance," the authors demonstrated that CSKD models exhibit significantly higher attention distances in later layers compared to standard DeiT. This empirical evidence confirms that the cumulative strategy successfully encourages the ViT to integrate information across the entire image rather than remaining tethered to local patches.
Qualitative heatmaps further support these findings, showing that CSKD-trained models focus more accurately on salient objects compared to baselines. The method also showed strong generalization on transfer learning tasks, notably achieving a +1.1% improvement on the iNaturalist 2019 dataset.
Conclusion
Cumulative Spatial Knowledge Distillation provides an elegant solution to the trade-off between local inductive bias and global modeling capacity. By simplifying the distillation pipeline and introducing a dynamic "graduation" mechanism, CSKD enables Vision Transformers to surpass the limitations of their CNN teachers. While the method introduces some training overhead, its performance gains and improved global attention make it a robust choice for efficient ViT training.
Summary updated