Abstract
Knowledge distillation is an effective method for training lightweight models, but it introduces a significant amount of computational overhead to the training cost, as the method requires acquiring teacher supervisions on training samples. This additional cost -- called distillation cost -- is most pronounced when we employ large-scale teacher models such as vision transformers (ViTs). We present MaskedKD, a simple yet effective strategy that can significantly reduce the cost of distilling ViTs without sacrificing the prediction accuracy of the student model. Specifically, MaskedKD diminishes the cost of running teacher at inference by masking a fraction of image patch tokens fed to the teacher, and therefore skipping the computations required to process those patches. The mask locations are selected to prevent masking away the core features of an image that the student model uses for prediction. This mask selection mechanism operates based on some attention score of the student model, which is already computed during the student forward pass, and thus incurs almost no additional computation. Without sacrificing the final student accuracy, MaskedKD dramatically reduces the amount of computations required for distilling ViTs. We demonstrate that MaskedKD can save up the distillation cost by $50\%$ without any student performance drop, leading to approximately $28\%$ drop in the overall training FLOPs.
Introduction
Knowledge Distillation (KD) is the gold standard for training small, efficient models (students) to mimic the performance of massive, pre-trained models (teachers). However, there is a catch: the "supervision cost." To train the student, you have to run the massive teacher model on every single training image, often multiple times if you're using data augmentation. For giant models like ViT-G, this can consume tens of thousands of TPU-days.
Previous attempts to fix this involved pre-computing the teacher’s labels, but this fails when you use modern data augmentations like Mixup, which change the image on every iteration. MaskedKD addresses this by asking: Can we make the teacher do less work by only showing it part of the image?
Methods
MaskedKD is a framework that uses masking to reduce the teacher's workload. The process works in four distinct steps:
- Student Forward Pass: The student model looks at the full image.
- Saliency Scoring: The framework extracts the attention scores from the student’s final layer. Specifically, it looks at which patches the student's "class token" is paying attention to.
- Teacher Masking: Based on these scores, the "least important" patches (according to the student) are masked out. The teacher model is then fed this sparse, masked image.
- Distillation: The student is trained to match the teacher’s predictions. Because the teacher is processing fewer tokens, it skips a massive amount of computation.
Crucially, masking at the input level is more effective for distillation than "token pruning" in the middle of the network. While masking the student model usually hurts performance, masking the teacher actually helps.
Key Insights and Ideas
- The Student-Guided Curriculum: Student-guided masking acts as a natural curriculum. In early training, the student's attention is noisy, forcing the teacher to provide noisier, easier-to-mimic supervision. As the student improves, it focuses on core features, and the teacher provides specialized supervision on those specific areas.
- Input Masking > Intermediate Pruning: For supervision quality, it is better to drop tokens right at the start rather than pruning them as they move through layers.
- Teacher Assistants by Proxy: A masked teacher behaves similarly to a "Teacher Assistant." Masking effectively "downgrades" the teacher's capacity just enough to make it a better coach for the student.
Key Results
MaskedKD consistently saves 25% to 50% of teacher FLOPs. In some extreme cases, 75% of the patches could be masked with only a negligible drop in accuracy. Surprisingly, student models often performed better than those trained with standard KD. For example, a DeiT-Ti student distilled from a DeiT-S teacher reached 75.2% accuracy (up from 75.0%) while using 51% fewer teacher FLOPs.
The method also showed cross-domain success, successfully reducing costs for audio classification (ESC-50) and dense tasks like segmentation (ADE20k) and detection (COCO) without sacrificing performance.
Limitations and Future Work
MaskedKD is currently specialized for Transformer-based models because the masking logic relies on attention scores. Applying this to pure Convolutional Neural Networks (CNNs) would require a different saliency mechanism. Additionally, because the model chooses which features are "important," there is a risk of strengthening biases present in the training data. Future work involves generalizing this framework to a broader range of architectures and investigating bias mitigation.
Summary updated