Masked Distillation with Receptive Tokens

Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, Chang Xu

Introduction

MasKD (Masked Distillation with Receptive Tokens) is a novel knowledge distillation framework that moves beyond coarse, bounding-box-based feature imitation. By using learnable "receptive tokens" to generate fine-grained, pixel-level masks, it identifies the most informative regions for a student model to learn from a teacher. This approach leads to state-of-the-art performance in dense prediction tasks like object detection and semantic segmentation.

The Problem: Beyond Bounding Boxes

Knowledge Distillation (KD) typically involves a large "teacher" model guiding a smaller, more efficient "student" model. In dense prediction tasks like object detection, a common approach is to have the student mimic the teacher’s intermediate feature maps.

However, the authors identify a major flaw in current methods: not all pixels are created equal. Standard methods either treat every pixel with equal weight (which introduces noise) or use ground-truth bounding boxes to define "important" regions.

Comparison of bounding box annotations versus fine-grained activation maps.

The problem with relying on bounding boxes is twofold:

  1. They are coarse: A box often contains a significant amount of irrelevant background noise.
  2. They ignore valuable context: Valuable "background" pixels outside the box might actually help the model distinguish objects and understand scene context.

The core challenge is finding a way to localize "Pixels of Interest" (PoIs) at a fine-grained level without relying on manual annotations or task-specific priors.

Methodology: The MasKD Framework

To address these limitations, the authors propose a two-stage approach called MasKD.

The MasKD architecture, illustrating how receptive tokens generate masks to guide feature distillation between the teacher and student models.
  1. Learning Masks with Receptive Tokens: The framework introduces "receptive tokens"—learnable embeddings that interact with the teacher's feature maps via a pixel-wise attention mechanism. This interaction generates a set of masks. To ensure these masks are useful, they are trained to minimize the teacher's original task loss while the teacher's weights remain frozen.
  2. Mask Diversity & Weighting: To prevent all tokens from redundantly focusing on the exact same pixels, a Dice loss is utilized to encourage diversity among the generated masks. Additionally, a small "mask weighting module" (a simple convolutional block) predicts the importance of each mask for a specific image, allowing the distillation process to prioritize foreground-like features over empty or noisy ones.
  3. Student-Aware Customization: Recognizing the "capacity gap" between a highly complex teacher and a smaller student, the authors refine the masks using the student's own features. Distillation only occurs on pixels that both the teacher and student deem important. This prevents the student from being overwhelmed by complex teacher features it lacks the capacity to learn.

Key Insights

  • The "Box" is a Lie: The paper demonstrates that the most "activated" regions for a teacher model are often much smaller and more irregular than ground-truth bounding boxes. Forcing a student to mimic the entire box forces it to mimic noise.
  • Background Matters: Some "background" pixels contain discriminative information. MasKD's learnable tokens can capture these regions dynamically, whereas rigid box-based methods discard them entirely.
  • Dynamic Distillation: Unlike static masks, MasKD’s masks are instance-dependent. The importance of a specific pixel changes depending on the specific image being processed.

Key Results

The researchers evaluated MasKD on three primary benchmarks: MS COCO (object detection/instance segmentation), Cityscapes (urban semantic segmentation), and ImageNet (image classification). MasKD consistently outperformed previous state-of-the-art distillation methods:

  • Object Detection (COCO): A Faster RCNN-R50 student improved by 2.4 AP (from 38.4 to 40.8). When using a stronger teacher (Cascade Mask RCNN), the gain reached an impressive 3.8 AP.
  • Semantic Segmentation (Cityscapes): A DeepLabV3-R18 student trained from scratch saw a massive jump of 8.78 mIoU (from 65.17 to 73.95), significantly beating the previous best method, CIRKD.
  • Generalization (ImageNet): Proving the method isn't just for dense prediction, MasKD improved the baseline mimicry approach by 0.55% on standard image classification.

Limitations and Future Work

While highly effective, the framework does have limitations. The two-stage training process adds an extra step to the pipeline, although the authors note the computational cost of the mask-learning stage is negligible. Furthermore, the method introduces a few new hyperparameters, such as the number of tokens and a loss balancing factor, though a moderate number of tokens (around 6) was found to work well across most tasks.

For future work, the authors suggest exploring how these receptive tokens behave in even more complex environments, such as panoptic segmentation or video-based detection tasks.

Summary updated

Abstract

Distilling from the feature maps can be fairly effective for dense prediction tasks since both the feature discriminability and localization information can be well transferred. However, not every pixel contributes equally to the performance, and a good student should learn from what really matters to the teacher. In this paper, we introduce a learnable embedding dubbed receptive token to locate the pixels of interests (PoIs) in the feature map, with a distillation mask generated via pixel-wise attention. Then the masked distillation will be performed via the pixel-wise reconstruction. In this way, a distillation mask refers to a pattern of pixel dependencies. We thus adopt multiple receptive tokens to investigate more sophisticated and informative pixel dependencies within feature maps to enhance the distillation. To obtain a group of masks, the receptive tokens are learned via the regular task loss but with teacher fixed, and we also leverage a Dice loss to enrich the diversity of obtained masks. Our method dubbed MasKD is simple and practical, and needs no priors of ground-truth labels, which can apply to various dense prediction tasks. Experiments show that our MasKD can achieve state-of-the-art performance consistently on object detection and semantic segmentation benchmarks.
BibTeX
@inproceedings{huang2023masked,
  author = {Tao Huang and Yuan Zhang and Shan You and Fei Wang and Chen Qian and Jian Cao and Chang Xu},
  title = {{Masked Distillation with Receptive Tokens}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=mWRngkvIki3},
}
Figure