Masked Autoencoders Are Stronger Knowledge Distillers

Shanshan Lao, Guanglu Song, Boxiao Liu, Yu Liu, Yujiu Yang

Introduction

Knowledge Distillation (KD) has emerged as a cornerstone technique for transferring knowledge from high-capacity "teacher" models to lightweight "student" models. While effective, feature-based KD often encounters a significant bottleneck in dense visual tasks like object detection and semantic segmentation: information redundancy.

Empirical observations suggest that a substantial portion of spatial features in a teacher network—approximately 72%—are "simple" and easily mimicked. This abundance of easy-to-learn information can overshadow the critical, complex knowledge required for superior performance. The authors demonstrate that up to 30% of a teacher's features can be randomly masked without performance degradation, highlighting the extent of this redundancy. Traditional KD methods, by treating all pixels with equal importance, often lead the student to learn redundant information while failing to capture the intricate underlying structures necessary for true mastery.

Masked Knowledge Distillation (MKD)

To address the redundancy bottleneck, the authors introduce Masked Knowledge Distillation (MKD). This approach leverages the principles of Masked Autoencoders (MAE) to create a more challenging and informative distillation task.

Overview of the Masked Knowledge Distillation (MKD) framework, illustrating the asymmetric input strategy and the adaptive decoder architecture.

The MKD framework consists of several key components designed to force the student to learn deeper relationships:

  1. Asymmetric Masking: The student model receives a partially masked input image (random patches are hidden), while the teacher model processes the full, uncorrupted image. This forces the student to infer missing information rather than simply copying visible pixels.
  2. Masked Convolutions: To prevent information leakage from visible to masked regions during encoding, standard convolutions are replaced with specialized Masked Convolutions. This ensures the student relies on broader context and learned representations rather than local pixel interpolation.
  3. Adaptive Decoder: A specialized decoder is employed to reconstruct the teacher's multi-scale features. It includes a Spatial Alignment Module (SAM) for resolution uniformity, a Transformer Decoder using "mask tokens" to capture long-range dependencies, and a Spatial Recovery Module (SRM) to map features back to their original scales for loss calculation.

By forcing the student to reconstruct the entire feature map (both masked and unmasked regions) from a corrupted input, MKD encourages the learning of robust, high-level structural representations that are often missed by traditional distillation.

Experimental Results

MKD was evaluated across major benchmarks including MS COCO, Cityscapes, and ImageNet. The results consistently demonstrate that MKD sets a new state-of-the-art for feature-based distillation.

Feature visualization comparing MKD with other distillation methods. MKD more accurately reconstructs the teacher's focus on critical object regions.

Key performance highlights include:

  • Object Detection: On MS COCO, MKD improved RetinaNet (ResNet-18) from 33.4 to 37.5 mAP. Notably, a Faster R-CNN (ResNet-50) student achieved 41.1 mAP, surpassing its ResNet-101 teacher (39.8 mAP).
  • Semantic Segmentation: On Cityscapes, MKD delivered a significant improvement of 4.85 mIoU for the PspNet-Res18 model, significantly closing the gap with the teacher.
  • Generalization: The masking task acts as a powerful regularizer. In several configurations, the student model trained with MKD actually outperformed the teacher model it was supposed to be mimicking, suggesting superior generalization.

Conclusion

Masked Knowledge Distillation effectively overcomes the information redundancy inherent in traditional KD. By transforming the distillation process into a reconstruction task, MKD forces student models to capture the essential structural knowledge of the teacher. While it introduces some training-time complexity due to the adaptive decoder, the resulting models are significantly more potent without any additional inference cost, making it a highly effective strategy for deploying high-performance lightweight models in real-world applications.

Summary updated

Abstract

Knowledge distillation (KD) has shown great success in improving student’s performance by mimicking the intermediate output of the high-capacity teacher in fine-grained visual tasks, e.g. object detection. This paper proposes a technique called Masked Knowledge Distillation (MKD) that enhances this process using a masked autoencoding scheme. In MKD, random patches of the input image are masked, and the corresponding missing feature is recovered by forcing it to imitate the output of the teacher. MKD is based on two core designs. First, using the student as the encoder, we develop an adaptive decoder architecture, which includes a spatial alignment module that operates on the multi-scale features in the feature pyramid network (FPN) [20], a simple decoder, and a spatial recovery module that mimics the teacher’s output from the latent representation and mask tokens. Second, we introduce the masked convolution in each convolution block to keep the masked patches unaffected by others. By coupling these two designs, we can further improve the completeness and effectiveness of teacher knowledge learning. We conduct extensive experiments on different architectures with object detection and semantic segmentation. The results show that all the students can achieve further improvements compared to the conventional KD. Notably, we establish the new state-of-the-art results by boosting RetinaNet ResNet-18, and ResNet-50 from 33.4 to 37.5 mAP, and 37.4 to 41.5 mAP, respectively.
BibTeX
@inproceedings{lao2023masked,
  author = {Shanshan Lao and Guanglu Song and Boxiao Liu and Yu Liu and Yujiu Yang},
  title = {{Masked Autoencoders Are Stronger Knowledge Distillers}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00587},
}
Figure