DFMSD: Dual Feature Masking Stage-wise Knowledge Distillation for Object Detection

Zhourui Zhang, Jun Li, Zhijian Wu, Jifeng Shen, Jianhua XuarXiv 2024 · arXiv · OpenReview

Abstract

In recent years, current mainstream feature masking distillation methods mainly function by reconstructing selectively masked regions of a student network from the feature maps of a teacher network. In these methods, attention mechanisms can help to identify spatially important regions and crucial object-aware channel clues, such that the reconstructed features are encoded with sufficient discriminative and representational power similar to teacher features. However, previous feature-masking distillation methods mainly address homogeneous knowledge distillation without fully taking into account the heterogeneous knowledge distillation scenario. In particular, the huge discrepancy between the teacher and the student frameworks within the heterogeneous distillation paradigm is detrimental to feature masking, leading to deteriorating reconstructed student features. In this study, a novel dual feature-masking heterogeneous distillation framework termed DFMSD is proposed for object detection. More specifically, a stage-wise adaptation learning module is incorporated into the dual feature-masking framework, and thus the student model can be progressively adapted to the teacher models for bridging the gap between heterogeneous networks. Furthermore, a masking enhancement strategy is combined with stage-wise learning such that object-aware masking regions are adaptively strengthened to improve feature-masking reconstruction. In addition, semantic alignment is performed at each Feature Pyramid Network (FPN) layer between the teacher and the student networks for generating consistent feature distributions. Our experiments for the object detection task demonstrate the promise of our approach, suggesting that DFMSD outperforms both the state-of-the-art heterogeneous and homogeneous distillation methods.

Introduction

Knowledge Distillation (KD) has become a cornerstone technique for deploying high-performance computer vision models on resource-constrained devices. By training a lightweight "student" model to mimic a larger, more complex "teacher," researchers can achieve significant performance gains without increasing inference costs. However, most existing methods focus on homogeneous distillation, where the teacher and student share similar architectures (e.g., both are Convolutional Neural Networks).

When faced with heterogeneous distillation—such as transferring knowledge from a Vision Transformer (ViT) like the Swin Transformer to a CNN like ResNet—a significant "semantic gap" emerges. The structural differences in how these models process information make direct feature mimicry difficult, often leading to stagnant performance in the student model. The proposed DFMSD (Dual Feature Masking Stage-wise Knowledge Distillation) framework addresses this gap through a multi-stage, frequency-aware approach.

Methodology

The DFMSD framework introduces three core components designed to bridge the architectural divide between heterogeneous networks.

Stage-wise Adaptation Learning (SAL)

One of the primary hurdles in heterogeneous distillation is the complexity of the teacher's feature representations. Forcing a small student to immediately reconstruct the features of a powerful "super-teacher" can be counterproductive. SAL introduces a curriculum-based learning process where the student first learns from a "tutor" (a weaker teacher, such as Swin-T) to build a foundation. Once the student has adapted to this intermediate representation, it progresses to learning from the stronger teacher (e.g., Swin-S). This step-by-step adaptation makes the final knowledge transfer significantly more effective.

Masking Enhancement (ME) Module

Object detection performance is often limited by the model's ability to handle objects at different scales. The ME module utilizes frequency-domain data augmentation to sharpen the student's focus. By adaptively applying Gaussian noise to images dominated by small objects (enhancing high-frequency details) or random cropping to images with large objects (enhancing low-frequency clues), the module forces the student to better identify and reconstruct masked regions across all scales.

Semantic Feature Alignment (SFA) Module

Because teacher and student architectures differ, their raw feature distributions are often incompatible. Instead of attempting to match exact pixel values, the SFA module uses Pearson correlation coefficients to align the distributions of feature maps at each layer of the Feature Pyramid Network (FPN). This ensures the student captures the same relative object-aware patterns as the teacher, focusing on the relationships between features rather than their absolute values.

Experimental Evaluation

The framework was evaluated on the MS COCO dataset, a standard benchmark for object detection. The authors tested various combinations of teachers (Swin Transformer, Cascade Mask R-CNN) and students (Faster R-CNN, RetinaNet, FCOS).

Quantitative Results

DFMSD showed consistent improvements across various architectures:

  • ViT to CNN: A Faster R-CNN student (baseline 38.4% mAP) reached 43.1% mAP (+4.7%) when distilled from Swin-T and Swin-S teachers. This outperformed the previous state-of-the-art method (DMKD) by 0.8%.
  • CNN to CNN: When distilling from a Cascade Mask R-CNN to a RetinaNet, DFMSD achieved 40.1% mAP, a 2.7% improvement over the baseline.

Qualitative Analysis

Visualizing the feature heatmaps reveals that DFMSD produces much more concentrated and accurate object-aware representations compared to standard detectors. As shown in the comparison below, the DFMSD-enhanced model focuses more precisely on the object of interest, reducing background noise and improving localization.

Visual comparison of feature heatmaps across different detection models. (a) Faster R-CNN, (b) RetinaNet, (c) FCOS, and (d) DFMSD (Ours). Note how the DFMSD heatmap is more precisely aligned with the object's body.

Conclusion

DFMSD effectively bridges the structural gap in heterogeneous knowledge distillation for object detection. By employing a stage-wise "tutor" approach and frequency-aware masking enhancement, it enables lightweight models to capture the sophisticated representational power of large-scale Transformers. While the stage-wise approach increases training time, the resulting performance gains and generalizability across anchor-based and anchor-free detectors make it a powerful tool for model compression. Future work will investigate the application of these principles to other tasks, such as semantic and instance segmentation.

Summary updated

Abstract

In recent years, current mainstream feature masking distillation methods mainly function by reconstructing selectively masked regions of a student network from the feature maps of a teacher network. In these methods, attention mechanisms can help to identify spatially important regions and crucial object-aware channel clues, such that the reconstructed features are encoded with sufficient discriminative and representational power similar to teacher features. However, previous feature-masking distillation methods mainly address homogeneous knowledge distillation without fully taking into account the heterogeneous knowledge distillation scenario. In particular, the huge discrepancy between the teacher and the student frameworks within the heterogeneous distillation paradigm is detrimental to feature masking, leading to deteriorating reconstructed student features. In this study, a novel dual feature-masking heterogeneous distillation framework termed DFMSD is proposed for object detection. More specifically, a stage-wise adaptation learning module is incorporated into the dual feature-masking framework, and thus the student model can be progressively adapted to the teacher models for bridging the gap between heterogeneous networks. Furthermore, a masking enhancement strategy is combined with stage-wise learning such that object-aware masking regions are adaptively strengthened to improve feature-masking reconstruction. In addition, semantic alignment is performed at each Feature Pyramid Network (FPN) layer between the teacher and the student networks for generating consistent feature distributions. Our experiments for the object detection task demonstrate the promise of our approach, suggesting that DFMSD outperforms both the state-of-the-art heterogeneous and homogeneous distillation methods.
BibTeX
@misc{zhang2024dfmsd,
  author = {Zhourui Zhang and Jun Li and Zhijian Wu and Jifeng Shen and Jianhua Xu},
  title = {{DFMSD: Dual Feature Masking Stage-wise Knowledge Distillation for Object Detection}},
  year = {2024},
  eprint = {2407.13147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.13147},
}
Figure