Masked Distillation with Receptive Tokens

Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, Chang Xu

Abstract

Distilling from feature maps can be highly effective for dense prediction tasks, as both feature discriminability and localization priors are transferred. However, not every pixel contributes equally to performance; a student model should ideally learn from what truly matters to the teacher. In this paper, we introduce a learnable embedding called a receptive token to localize Pixels of Interest (PoIs) in the feature map, using a distillation mask generated via pixel-wise attention. Distillation is then performed on the mask through pixel-wise reconstruction. In this framework, a distillation mask indicates a pattern of pixel dependencies within the teacher's feature maps. We adopt multiple receptive tokens to investigate sophisticated pixel dependencies, further enhancing the distillation. These tokens are learned via regular task loss with a fixed teacher, and a Dice loss is leveraged to enrich mask diversity. Our method, MasKD, is simple, practical, and requires no task-specific priors. Experiments demonstrate that MasKD consistently achieves state-of-the-art performance on object detection and semantic segmentation benchmarks.

Introduction

In Knowledge Distillation (KD), the goal is to transfer the expertise of a heavy, high-performing "teacher" model into a lightweight "student" model suitable for deployment on edge devices. For dense prediction tasks—such as object detection or semantic segmentation where decisions are made at the pixel level—standard distillation often struggles.

The core challenge is that not all pixels are created equal. Traditional methods typically attempt to mimic the entire feature map, which often includes significant background noise, or they rely on manual "priors" like ground-truth bounding boxes to guide the student's attention. However, bounding boxes are coarse; they frequently include irrelevant background pixels and may miss subtle, discriminative features outside the box that the teacher relies on. MasKD addresses this by asking: Can we let the model learn for itself which pixels actually matter for the task?

The MasKD Framework

MasKD introduces a smarter way for students to learn by using learnable "receptive tokens" to automatically identify and prioritize the most informative regions of an image.

The overall framework of MasKD, illustrating how receptive tokens generate masks to guide feature distillation between teacher and student backbones.

The framework operates in a two-stage process:

  1. Learning Receptive Tokens: The system introduces receptive tokens—learnable embeddings that act as probes. These tokens calculate pixel-wise attention against the teacher’s feature maps to generate a set of masks. To ensure these masks capture a wide variety of features, a Dice loss (diversity loss) is applied, forcing different tokens to focus on distinct Pixels of Interest (PoIs).
  2. Mask Weighting and Customization: Not every learned mask is equally important. A dedicated module assigns weights to these masks so the student prioritizes critical regions (e.g., a small object vs. a large patch of sky).
  3. Student-Aware Refinement: Uniquely, MasKD customizes the masks using the student's own features. This prevents the student from wasting capacity trying to reconstruct "hard-to-learn" noise or features irrelevant to its specific architecture, focusing instead on the intersection of what the teacher knows and what the student can realistically grasp.

Visualizing Pixels of Interest

A key insight of this research is that the "activated regions" the teacher uses to make decisions are often much finer than standard bounding box annotations. By moving beyond box-based priors, MasKD identifies pixel-level dependencies that manual heuristics miss.

Visualization of the evolution of learned masks across training stages (0-4) compared to standard ground-truth annotations. Note how the masks refine to focus on specific object parts.

As shown in the visualization, the receptive tokens evolve to capture sophisticated dependencies. One token might focus on object boundaries, while another focuses on internal textures. This diversity allows the student to inherit a more nuanced understanding of the scene than simple feature mimicking would allow.

Experimental Results

The researchers evaluated MasKD across several major benchmarks, including MS COCO for object detection, Cityscapes for urban semantic segmentation, and ImageNet for classification. MasKD achieved significant performance gains across various architectures:

  • Object Detection: On MS COCO, MasKD improved a Faster R-CNN (ResNet-50) student by 2.4 AP, outperforming previous state-of-the-art methods like GID and FGD.
  • Semantic Segmentation: On Cityscapes, it boosted a DeepLabV3 (ResNet-18) student by 2.79 mIoU.
  • Scaling with Strong Teachers: Even when paired with a massive Cascade Mask R-CNN (ResNeXt-101) teacher, MasKD successfully transferred knowledge, yielding a 3.8 AP gain over the student baseline.

Conclusion and Future Work

MasKD demonstrates that learnable receptive tokens provide a powerful, automated alternative to manual distillation priors. By focusing on the most informative pixels, student models can achieve higher accuracy without increased inference-time complexity.

While highly effective, the current implementation uses a two-stage process (mask learning followed by distillation). Future work could explore integrating these into a seamless, single-stage training pipeline. Additionally, while MasKD excels in visual tasks, exploring the behavior of receptive tokens in non-visual domains—such as natural language processing or signal analysis—remains a promising avenue for research.

Summary updated

Abstract

Distilling from the feature maps can be fairly effective for dense prediction tasks since both the feature discriminability and localization priors can be well transferred. However, not every pixel contributes equally to the performance, and a good student should learn from what really matters to the teacher. In this paper, we introduce a learnable embedding dubbed receptive token to localize those pixels of interests (PoIs) in the feature map, with a distillation mask generated via pixel-wise attention. Then the distillation will be performed on the mask via pixel-wise reconstruction. In this way, a distillation mask actually indicates a pattern of pixel dependencies within feature maps of teacher. We thus adopt multiple receptive tokens to investigate more sophisticated and informative pixel dependencies to further enhance the distillation. To obtain a group of masks, the receptive tokens are learned via the regular task loss but with teacher fixed, and we also leverage a Dice loss to enrich the diversity of learned masks. Our method dubbed MasKD is simple and practical, and needs no priors of tasks in application. Experiments show that our MasKD can achieve state-of-the-art performance consistently on object detection and semantic segmentation benchmarks. Code is available at: https://github.com/hunto/MasKD .
BibTeX
@inproceedings{huang2023masked,
  author = {Tao Huang and Yuan Zhang and Shan You and Fei Wang and Chen Qian and Jian Cao and Chang Xu},
  title = {{Masked Distillation with Receptive Tokens}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=mWRngkvIki3},
}
Figure