DMKD: Improving Feature-based Knowledge Distillation for Object Detection Via Dual Masking Augmentation

Guang Yang, Yin Tang, Zhijian Wu, Jun Li, Jianhua Xu, Xili Wan

Abstract

Recent mainstream masked distillation methods function by reconstructing selectively masked areas of a student network from the feature map of its teacher counterpart. In these methods, the masked regions need to be properly selected, such that reconstructed features encode sufficient discrimination and representation capability like the teacher feature. However, previous masked distillation methods only focus on spatial masking, making the resulting masked areas biased towards spatial importance without encoding informative channel clues. In this study, we devise a Dual Masked Knowledge Distillation (DMKD) framework which can capture both spatially important and channel-wise informative clues for comprehensive masked feature reconstruction. More specifically, we employ dual attention mechanism for guiding the respective masking branches, leading to reconstructed feature encoding dual significance. Furthermore, fusing the reconstructed features is achieved by self-adjustable weighting strategy for effective feature distillation. Our experiments on object detection task demonstrate that the student networks achieve performance gains of 4.1% and 4.3% with the help of our method when RetinaNet and Cascade Mask R-CNN are respectively used as the teacher networks, while outperforming the other state-of-the-art distillation methods.

Introduction

Knowledge Distillation (KD) is a critical technique for deploying high-performance models on resource-constrained devices by training a compact "student" network to emulate a larger "teacher" network. In the context of object detection, masked distillation has emerged as a powerful paradigm. This approach involves masking portions of the student's feature maps and requiring the student to reconstruct the teacher's original, unmasked features.

However, existing masked distillation methods primarily focus on spatial masking—identifying which pixels or regions are most salient. This oversight ignores the channel dimension, where different channels in modern neural networks encode distinct semantic information (e.g., textures, edges, or specific object parts). By neglecting channel-wise clues, previous methods struggle to capture the full "object-aware" context necessary for complex detection tasks.

Methodology

The proposed Dual Masked Knowledge Distillation (DMKD) framework addresses these limitations by capturing both spatially salient and channel-wise informative features. The architecture is divided into several key components:

  1. Dual Attention Generation: The teacher's feature maps are used to generate a spatial attention map ($R^{H \times W \times 1}$) and a channel attention map ($R^{1 \times 1 \times C}$).
  2. Attention-Guided Masking: Instead of random masking, DMKD uses these attention maps to mask the most significant regions and channels, forcing the student to reconstruct the most critical information.
  3. Specialized Reconstruction Blocks:
    • Conv-based Generation: Utilizes convolutional layers to capture local spatial context for spatial reconstruction.
  4. MLP-based Generation: Uses a Multi-Layer Perceptron (MLP) to model inter-channel relationships independently of spatial location, which is more effective for channel-wise clues.
  5. Self-Adjustable Fusion: The reconstructed features are combined using learnable weights ($\alpha$ and $\beta$), allowing the model to dynamically balance spatial and channel information during training.
The DMKD framework architecture, showcasing the dual masking branches and specialized reconstruction blocks.

Experimental Results

The researchers evaluated DMKD on the MS COCO 2017 dataset using various detector architectures, including RetinaNet, RepPoints, and Cascade Mask R-CNN. In all experiments, a ResNeXt101 teacher was used to distill knowledge into a ResNet50 student.

DMKD consistently outperformed state-of-the-art (SOTA) methods like FGD, MGD, and AMD. For instance, it improved RetinaNet's mAP by 4.1% (from 37.4% to 41.5%) and Cascade Mask R-CNN's mAP by 4.3% (reaching 42.7%). Notably, DMKD is a training-time augmentation; it adds no computational overhead during inference.

Qualitative comparison of attention maps between MGD, AMD, and DMKD. DMKD shows superior focus on the object, correlating with its higher mAP.

Conclusion

DMKD demonstrates that incorporating channel-wise information into masked distillation significantly enhances the performance of student models in object detection. By leveraging dual attention mechanisms and specialized reconstruction blocks, the framework captures a more comprehensive representation of the teacher's knowledge. Future work may extend this dual-masking approach to other dense prediction tasks like semantic or instance segmentation.

Summary updated

Abstract

Recent mainstream masked distillation methods function by reconstructing selectively masked areas of a student network from the feature map of its teacher counterpart. In these methods, the masked regions need to be properly selected, such that reconstructed features encode sufficient discrimination and representation capability like the teacher feature. However, previous masked distillation methods only focus on spatial masking, making the resulting masked areas biased towards spatial importance without encoding informative channel clues. In this study, we devise a Dual Masked Knowledge Distillation (DMKD) framework which can capture both spatially important and channel-wise informative clues for comprehensive masked feature reconstruction. More specifically, we employ dual attention mechanism for guiding the respective masking branches, leading to reconstructed feature encoding dual significance. Furthermore, fusing the reconstructed features is achieved by self-adjustable weighting strategy for effective feature distillation. Our experiments on object detection task demonstrate that the student networks achieve performance gains of 4.1% and 4.3% with the help of our method when RetinaNet and Cascade Mask R-CNN are respectively used as the teacher networks, while outperforming the other state-of-the-art distillation methods.
BibTeX
@misc{yang2023dmkd,
  author = {Guang Yang and Yin Tang and Zhijian Wu and Jun Li and Jianhua Xu and Xili Wan},
  title = {{DMKD: Improving Feature-based Knowledge Distillation for Object Detection Via Dual Masking Augmentation}},
  year = {2023},
  eprint = {2309.02719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.02719},
}
Figure