AMD: Adaptive Masked Distillation for Object Detection

Guang Yang, Yin Tang, Jun Li, Jianhua Xu, Xili Wan

TL;DR

AMD (Adaptive Masked Distillation) is a new framework that boosts the performance of lightweight "student" object detectors by having them reconstruct specifically important regions of a "teacher" model's feature maps. By replacing random masking with attention-guided spatial masking and adding channel-wise adaptivity, AMD allows student models to achieve state-of-the-art accuracy, sometimes even surpassing their much larger teachers.

What Problem Does This Paper Address?

Deep Convolutional Neural Networks (CNNs) are powerful but computationally expensive, making them difficult to deploy on mobile or edge devices. Knowledge Distillation (KD) is a popular solution where a small "student" model learns to mimic a large "teacher" model.

In object detection, recent "feature-based" distillation methods have found success by masking parts of the student's feature map and forcing the student to reconstruct the teacher's corresponding features. However, existing methods like MGD (Masked Generative Distillation) use random masking. The authors argue this is inefficient: why force a student to spend its limited capacity reconstructing unimportant background pixels? Furthermore, most methods ignore the importance of specific feature channels, which are crucial for detecting objects of varying scales.

Methods

The authors propose the Adaptive Masked Distillation (AMD) framework, which introduces two primary innovations:

  1. Spatially Adaptive Feature Masking (Ada-Mask): Instead of masking pixels at random, AMD calculates a spatial attention map from the teacher’s features. It identifies the "Region-of-Interest" (RoI) where the teacher is focusing. The student’s feature map is then masked in these high-importance areas, forcing the student to learn how to reconstruct the most critical object-aware information.
  2. Channel Adaptive Clues (Ada-Channel): Object detection is a dense prediction task where different feature channels often represent objects at different scales. AMD uses a Squeeze-and-Excitation (SE) block to extract channel-wise importance from the teacher. These "clues" are fused with the student's reconstructed features using a Hadamard product, helping the student become more "object-aware."

The final training objective is a combination of the original detection loss and a distillation loss that measures how well the student reconstructed the teacher's masked, channel-adapted features.

Datasets

The researchers evaluated AMD using the MS COCO 2017 benchmark, the industry standard for object detection.

  • Training: 120k images.
  • Testing/Validation: 5k images.
  • Metrics: Average Precision (AP) and Average Recall (AR), with specific focus on $AP_S$ (small objects), $AP_M$ (medium), and $AP_L$ (large).

Key Insights and Ideas

  • Attention over Randomness: The "aha moment" here is that not all pixels are created equal. By using the teacher's own attention as a guide, the student focuses its learning energy on the foreground objects that actually matter for detection.
  • Reconstruction is Better than Imitation: AMD reinforces the idea that reconstructing a teacher's missing information is a more powerful learning signal for a student than simply copying the teacher's output.
  • Small Object Gains: One of the most interesting findings was that the channel-adaptive design significantly improved the detection of small objects, a notorious pain point for lightweight models.
  • Surpassing the Teacher: In some configurations, the student model actually achieved a higher mAP than the teacher model it was learning from, suggesting that the distillation process can act as a powerful regularizer.

Key Results

AMD consistently outperformed previous state-of-the-art distillation methods like FGD and MGD across various architectures:

  • RetinaNet: The student (ResNet50) reached 41.3% mAP, surpassing the teacher (ResNeXt101) which had 41.0% mAP. This was a +3.9% boost over the baseline student.
  • RepPoints: The student improved from a baseline of 38.6% to 42.7% mAP (+4.1% gain).
  • Faster R-CNN: In a cross-framework scenario (using a Cascade Mask R-CNN teacher), the student reached 42.4% mAP, a +4.0% improvement over the baseline.
  • Efficiency: The ablation studies showed that both the Ada-Mask and Ada-Channel components are necessary; removing either one led to a measurable drop in performance, especially for small objects.

Limitations and Future Work

The authors note that while their spatial attention map identifies important regions, it currently lacks "information interaction" between different locations on the map. In other words, the mask treats each pixel's importance somewhat in isolation.

Future Work: The team plans to explore alternative strategies to enhance the interaction between different locations on the attention map, potentially using more complex attention mechanisms or transformers to further refine which regions the student should prioritize.

Summary updated

Abstract

As a general model compression paradigm, feature-based knowledge distillation allows the student model to learn expressive features from the teacher counterpart. In this paper, we mainly focus on designing an effective feature-distillation framework and propose a spatial-channel adaptive masked distillation (AMD) network for object detection. More specifically, in order to accurately reconstruct important feature regions, we first perform attention-guided feature masking on the feature map of the student network, such that we can identify the important features via spatially adaptive feature masking instead of random masking in the previous methods. In addition, we employ a simple and efficient module to allow the student network channel to be adaptive, improving its model capability in object perception and detection. In contrast to the previous methods, more crucial object-aware features can be reconstructed and learned from the proposed network, which is conducive to accurate object detection. The empirical experiments demonstrate the superiority of our method: with the help of our proposed distillation method, the student networks report 41.3%, 42.4%, and 42.7% mAP scores when RetinaNet, Cascade Mask-RCNN and RepPoints are respectively used as the teacher framework for object detection, which outperforms the previous state-of-the-art distillation methods including FGD and MGD.
BibTeX
@misc{yang2023amd,
  author = {Guang Yang and Yin Tang and Jun Li and Jianhua Xu and Xili Wan},
  title = {{AMD: Adaptive Masked Distillation for Object Detection}},
  year = {2023},
  eprint = {2301.13538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.13538},
}
Figure