TL;DR
Knowledge distillation from massive Vision Transformers (ViTs) is computationally expensive because the teacher model must evaluate millions of augmented training images. MaskedKD cuts this "supervision cost" by up to 50% by masking out non-essential image patch tokens before feeding them to the teacher. Guided by the student’s own attention map, this input-level masking requires no modifications to the teacher architecture and preserves—or even slightly improves—student accuracy.
What Problem Does This Paper Address?
While knowledge distillation is a standard recipe for training lightweight models, acquiring predictions from gigantic ViT teachers creates a massive computational bottleneck. Distilling a model like ViT-G on ImageNet can consume roughly $10^4$ TPUv3-days just running the teacher forward pass.
Pre-computing teacher predictions on static images fails in modern training pipelines because strong data augmentations (e.g., Mixup, RandAugment) require on-the-fly teacher evaluation. Furthermore, many state-of-the-art vision models are proprietary (accessible only as black-box inference APIs), preventing users from modifying internal layers or retraining the teacher. The authors set out to drastically reduce online teacher inference costs without sacrificing the student’s final downstream accuracy.
Methods
MaskedKD introduces a simple four-step distillation loop:
- Student Forward Pass on Full Image: The student model processes the unmasked image $x$.
-
Student-Informed Patch Saliency (SIPS): To decide which image patches to drop, MaskedKD computes an importance score for each patch using the student's last multi-head self-attention layer. Specifically, it sums the attention weights from the
[CLS]token to all image patch keys across all heads: $$\bar{a} = \sum_{h=1}^H a^{(h)}$$ Because this re-uses representations already computed during the student forward pass, SIPS introduces negligible compute ($< 28\text{k}$ FLOPs). -
Teacher Forward Pass on Masked Image: The patches with the lowest SIPS scores are removed, and only the top-$\kappa\%$ salient patches ($x_{\text{mask}}$) are converted into tokens and passed to the teacher. Masking at the input allows the teacher to skip calculations for omitted tokens across all subsequent layers without architectural modifications.
- Distillation: Standard distillation losses (such as KL divergence over logits or feature distance) match the student’s predictions on the full image to the teacher’s predictions on the masked image.
To prevent GPU idling caused by the teacher waiting on student attention scores, the pipeline uses micro-batch pipelining, processing small subsets sequentially so both models stay saturated.
Full Image ---> [ Student ViT ] ---> Student Logits
|
(SIPS Attention Map)
v
Masked Image --> [ Teacher ViT ] ---> Teacher Logits
| |
+---- Distillation <---+
Datasets
- ImageNet-1k: Primary benchmark for image classification and self-supervised distillation across multiple model scales (DeiT, CaiT, CLIP, MAE-initialized ViTs).
- ESC-50: An environmental sound classification dataset used to evaluate MaskedKD on Audio Spectrogram Transformers (AST).
- No new benchmarks were introduced.
Key Insights and Ideas
- Mask the teacher, never the student: In self-supervised learning (e.g., MAE), students receive masked inputs while teachers receive full views. In supervised distillation, doing this hurts accuracy even at low mask ratios. Conversely, masking the teacher retains or enhances supervision quality.
- The student guides the teacher: Masking patches randomly degrades distillation quality. Guiding the teacher's input using the student's own attention focus ensures the teacher predicts on the exact semantic regions the student is actively trying to learn, filtering out confusing background contexts.
- Input masking beats internal token merging: Compared to methods like Token Merging (ToMe) that drop or merge tokens progressively through internal layers, input-level masking works on black-box proprietary APIs and yields better supervision quality.
- Teaching Assistant effect: In large model capacity gaps (e.g., DeiT3-Base distilling into DeiT-Tiny), masking the teacher by 25–50% actually improved student accuracy over full-image distillation, behaving similarly to a calibrated "teaching assistant."
Key Results
- Up to 50% FLOPs Reduction: On ImageNet-1k, masking 25–50% of teacher input tokens maintained or improved student accuracy across multiple teacher-student pairings. For instance:
- DeiT-S $\rightarrow$ DeiT-Ti: Teacher FLOPs dropped by 51% (1770 to 866 PFLOPs) while student accuracy rose from 75.0% to 75.2%.
- DeiT3-B $\rightarrow$ DeiT-Ti: Teacher FLOPs dropped by 50% while student accuracy increased from 74.4% to 74.7% (and stayed at 74.7% even with 75% tokens removed).
- Flexibility Across Paradigms:
- Self-Supervised Learning (DINO): Masking the DINO teacher to 66% of patches improved linear probe accuracy on ImageNet from 73.7% to 74.2% while slashing teacher compute from 589 to 384 PFLOPs.
- Audio (AST on ESC-50): Safely pruned 17% of spectrogram patches without loss in accuracy (86.3% $\rightarrow$ 86.4%).
Limitations and Future Work
- Transformer-Specific: The mechanism relies entirely on patch tokenization and self-attention maps; it does not directly transfer to convolutional or recurrent architectures.
- Manual Masking Ratio Tuning: There is currently no adaptive mechanism to determine the optimal patch retention ratio $\kappa$ per task or domain. While 50% works well for vision, audio data showed performance drop-offs beyond 17% masking.
- Integration with Masked Pretraining: While compatible with DINO self-distillation, integrating MaskedKD cleanly into pure masked image modeling pre-training (like MAE) remains an open challenge.
Summary updated