Abstract
Distilling from feature maps can be highly effective for dense prediction tasks, as both feature discriminability and localization priors are transferred. However, not every pixel contributes equally to performance; a student model should ideally learn from what truly matters to the teacher. In this paper, we introduce a learnable embedding called a receptive token to localize Pixels of Interest (PoIs) in the feature map, using a distillation mask generated via pixel-wise attention. Distillation is then performed on the mask through pixel-wise reconstruction. In this framework, a distillation mask indicates a pattern of pixel dependencies within the teacher's feature maps. We adopt multiple receptive tokens to investigate sophisticated pixel dependencies, further enhancing the distillation. These tokens are learned via regular task loss with a fixed teacher, and a Dice loss is leveraged to enrich mask diversity. Our method, MasKD, is simple, practical, and requires no task-specific priors. Experiments demonstrate that MasKD consistently achieves state-of-the-art performance on object detection and semantic segmentation benchmarks.
Introduction
In Knowledge Distillation (KD), the goal is to transfer the expertise of a heavy, high-performing "teacher" model into a lightweight "student" model suitable for deployment on edge devices. For dense prediction tasks—such as object detection or semantic segmentation where decisions are made at the pixel level—standard distillation often struggles.
The core challenge is that not all pixels are created equal. Traditional methods typically attempt to mimic the entire feature map, which often includes significant background noise, or they rely on manual "priors" like ground-truth bounding boxes to guide the student's attention. However, bounding boxes are coarse; they frequently include irrelevant background pixels and may miss subtle, discriminative features outside the box that the teacher relies on. MasKD addresses this by asking: Can we let the model learn for itself which pixels actually matter for the task?
The MasKD Framework
MasKD introduces a smarter way for students to learn by using learnable "receptive tokens" to automatically identify and prioritize the most informative regions of an image.
The framework operates in a two-stage process:
- Learning Receptive Tokens: The system introduces receptive tokens—learnable embeddings that act as probes. These tokens calculate pixel-wise attention against the teacher’s feature maps to generate a set of masks. To ensure these masks capture a wide variety of features, a Dice loss (diversity loss) is applied, forcing different tokens to focus on distinct Pixels of Interest (PoIs).
- Mask Weighting and Customization: Not every learned mask is equally important. A dedicated module assigns weights to these masks so the student prioritizes critical regions (e.g., a small object vs. a large patch of sky).
- Student-Aware Refinement: Uniquely, MasKD customizes the masks using the student's own features. This prevents the student from wasting capacity trying to reconstruct "hard-to-learn" noise or features irrelevant to its specific architecture, focusing instead on the intersection of what the teacher knows and what the student can realistically grasp.
Visualizing Pixels of Interest
A key insight of this research is that the "activated regions" the teacher uses to make decisions are often much finer than standard bounding box annotations. By moving beyond box-based priors, MasKD identifies pixel-level dependencies that manual heuristics miss.
As shown in the visualization, the receptive tokens evolve to capture sophisticated dependencies. One token might focus on object boundaries, while another focuses on internal textures. This diversity allows the student to inherit a more nuanced understanding of the scene than simple feature mimicking would allow.
Experimental Results
The researchers evaluated MasKD across several major benchmarks, including MS COCO for object detection, Cityscapes for urban semantic segmentation, and ImageNet for classification. MasKD achieved significant performance gains across various architectures:
- Object Detection: On MS COCO, MasKD improved a Faster R-CNN (ResNet-50) student by 2.4 AP, outperforming previous state-of-the-art methods like GID and FGD.
- Semantic Segmentation: On Cityscapes, it boosted a DeepLabV3 (ResNet-18) student by 2.79 mIoU.
- Scaling with Strong Teachers: Even when paired with a massive Cascade Mask R-CNN (ResNeXt-101) teacher, MasKD successfully transferred knowledge, yielding a 3.8 AP gain over the student baseline.
Conclusion and Future Work
MasKD demonstrates that learnable receptive tokens provide a powerful, automated alternative to manual distillation priors. By focusing on the most informative pixels, student models can achieve higher accuracy without increased inference-time complexity.
While highly effective, the current implementation uses a two-stage process (mask learning followed by distillation). Future work could explore integrating these into a seamless, single-stage training pipeline. Additionally, while MasKD excels in visual tasks, exploring the behavior of receptive tokens in non-visual domains—such as natural language processing or signal analysis—remains a promising avenue for research.
Summary updated