Abstract
Self-Knowledge Distillation (SKD) leverages the student's own knowledge to create a virtual teacher for distillation when the pre-trained bulky teacher is not available. Whilst existing SKD approaches demonstrate gorgeous efficiency in single-label learning, to directly apply them to multi-label learning would suffer from dramatic degradation due to the following inherent imbalance: targets with unified labels but multifarious visual scales are crammed into one image, resulting in biased learning of major targets and disequilibrium of precision-recall. To address this issue, this paper proposes a novel SKD method for multi-label learning named Multi-label Self-knowledge Distillation (MSKD), incorporating three Spatial Decoupling mechanisms (i.e. Locality-SD (L-SD), Reconstruction-SD (R-SD), and Step-SD (S-SD)). L-SD exploits relational dark knowledge from regional outputs to amplify the model's perception of visual details. R-SD reconstructs global semantics by integrating regional outputs from local patches and leverages it to guide the model. S-SD aligns outputs of the same input at different steps, aiming to find a synthetical optimizing direction and avoid the overconfidence. In addition, MSKD combines our tailored loss named MBD for balanced distillation. Exhaustive experiments demonstrate that MSKD not only outperforms previous approaches but also effectively mitigates biased learning and equips the model with more robustness.
Introduction
Knowledge Distillation (KD) has traditionally relied on a "teacher-student" paradigm, where a large, pre-trained model guides a smaller one. Self-Knowledge Distillation (SKD) simplifies this by allowing a model to act as its own teacher, significantly reducing computational overhead. However, existing SKD methods are primarily optimized for single-label classification. When applied to Multi-Label Learning (MLL), these methods often fail to account for the "inherent imbalance" of natural scenes, where large, prominent objects dominate the model's attention, causing smaller or partially obscured targets to be overlooked.
The MSKD Framework
To overcome these limitations, researchers developed Multi-label Self-Knowledge Distillation (MSKD). The core innovation lies in three Spatial Decoupling (SD) mechanisms designed to force the model to perceive fine-grained details and disjoint patterns that are typically ignored in global views.
Spatial Decoupling Mechanisms
- Locality-SD (L-SD): This mechanism crops random patches from an image and zooms in on them. By comparing the "dark knowledge" (the statistical relationships between classes) found in these magnified patches to the corresponding regions in the original image, the model learns to recognize subtle visual details.
- Reconstruction-SD (R-SD): Utilizing Graph Propagation (GP), R-SD identifies "overachiever" labels—classes that appear much more strongly in a local patch than in the global view. It then reconstructs a virtual teacher that integrates these local insights, providing a more balanced global semantic representation.
- Step-SD (S-SD): To ensure training stability and prevent overfitting to noisy labels, S-SD aligns the model's current predictions with its predictions from the previous training step, providing a consistent optimization direction.
Multi-label Balanced Distillation (MBD) Loss
Standard distillation losses like KL-Divergence often suffer from "gradient confusion" in multi-label settings, where signals from negative samples (absent objects) overwhelm the signals from positive samples (present objects). The proposed MBD loss reformulates the distillation objective to ensure that positive and negative signals are balanced, preventing the model from becoming overly conservative in its predictions.
Key Insights and Results
The effectiveness of MSKD is most apparent when dealing with complex scenes containing multiple objects of varying scales. By forcing the model to distill knowledge from magnified local patches back into the global view, the model learns that "small" does not mean "unimportant."
As shown in the figure above, standard training (w/o MSKD) often misidentifies objects or misses them entirely when they are small or obscured—for instance, misidentifying a boat as a bus or train. MSKD significantly improves the confidence scores for the correct labels, demonstrating its ability to "see" beyond the dominant features.
Performance Benchmarks
MSKD was evaluated on Pascal-VOC 2007, MS-COCO, and MIRFLICKR across various architectures (ResNet34, MobileNet_v2, and Swin-Transformer):
- Pascal VOC: MSKD achieved a 2.84% mAP improvement over previous state-of-the-art methods using a ResNet34 backbone.
- Small Object Recognition: On Pascal VOC, MSKD showed up to a 12% improvement in recognizing the smallest targets compared to standard training.
- Robustness: Unlike many SKD methods that struggle with the complexity of MS-COCO, MSKD provided stable, significant gains (+0.76% mAP on COCO with ResNet34).
Conclusion
MSKD represents a significant step forward in bringing the efficiency of self-distillation to the multi-label domain. By explicitly addressing the spatial imbalance of multi-label images through decoupling and balanced loss functions, MSKD enables smaller models to achieve high-fidelity perception of complex scenes. Future work will explore extending these self-distillation principles to other multi-modal and temporal tasks like action recognition.
Summary updated