Abstract
Recent mainstream masked distillation methods function by reconstructing selectively masked areas of a student network from the feature map of its teacher counterpart. In these methods, the masked regions need to be properly selected, such that reconstructed features encode sufficient discrimination and representation capability like the teacher feature. However, previous masked distillation methods only focus on spatial masking, making the resulting masked areas biased towards spatial importance without encoding informative channel clues. In this study, we devise a Dual Masked Knowledge Distillation (DMKD) framework which can capture both spatially important and channel-wise informative clues for comprehensive masked feature reconstruction. More specifically, we employ dual attention mechanism for guiding the respective masking branches, leading to reconstructed feature encoding dual significance. Furthermore, fusing the reconstructed features is achieved by self-adjustable weighting strategy for effective feature distillation. Our experiments on object detection task demonstrate that the student networks achieve performance gains of 4.1% and 4.3% with the help of our method when RetinaNet and Cascade Mask R-CNN are respectively used as the teacher networks, while outperforming the other state-of-the-art distillation methods.
Introduction
Knowledge Distillation (KD) is a critical technique for deploying high-performance models on resource-constrained devices by training a compact "student" network to emulate a larger "teacher" network. In the context of object detection, masked distillation has emerged as a powerful paradigm. This approach involves masking portions of the student's feature maps and requiring the student to reconstruct the teacher's original, unmasked features.
However, existing masked distillation methods primarily focus on spatial masking—identifying which pixels or regions are most salient. This oversight ignores the channel dimension, where different channels in modern neural networks encode distinct semantic information (e.g., textures, edges, or specific object parts). By neglecting channel-wise clues, previous methods struggle to capture the full "object-aware" context necessary for complex detection tasks.
Methodology
The proposed Dual Masked Knowledge Distillation (DMKD) framework addresses these limitations by capturing both spatially salient and channel-wise informative features. The architecture is divided into several key components:
- Dual Attention Generation: The teacher's feature maps are used to generate a spatial attention map ($R^{H \times W \times 1}$) and a channel attention map ($R^{1 \times 1 \times C}$).
- Attention-Guided Masking: Instead of random masking, DMKD uses these attention maps to mask the most significant regions and channels, forcing the student to reconstruct the most critical information.
- Specialized Reconstruction Blocks:
- Conv-based Generation: Utilizes convolutional layers to capture local spatial context for spatial reconstruction.
- MLP-based Generation: Uses a Multi-Layer Perceptron (MLP) to model inter-channel relationships independently of spatial location, which is more effective for channel-wise clues.
- Self-Adjustable Fusion: The reconstructed features are combined using learnable weights ($\alpha$ and $\beta$), allowing the model to dynamically balance spatial and channel information during training.
Experimental Results
The researchers evaluated DMKD on the MS COCO 2017 dataset using various detector architectures, including RetinaNet, RepPoints, and Cascade Mask R-CNN. In all experiments, a ResNeXt101 teacher was used to distill knowledge into a ResNet50 student.
DMKD consistently outperformed state-of-the-art (SOTA) methods like FGD, MGD, and AMD. For instance, it improved RetinaNet's mAP by 4.1% (from 37.4% to 41.5%) and Cascade Mask R-CNN's mAP by 4.3% (reaching 42.7%). Notably, DMKD is a training-time augmentation; it adds no computational overhead during inference.
Conclusion
DMKD demonstrates that incorporating channel-wise information into masked distillation significantly enhances the performance of student models in object detection. By leveraging dual attention mechanisms and specialized reconstruction blocks, the framework captures a more comprehensive representation of the teacher's knowledge. Future work may extend this dual-masking approach to other dense prediction tasks like semantic or instance segmentation.
Summary updated