MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition

Chuanguang Yang, Zhulin An, Helong Zhou, Linhang Cai, Xiang Zhi, Jiwen Wu, Yongjun Xu, Qian Zhang

Abstract

Unlike the conventional Knowledge Distillation (KD), Self-KD allows a network to learn knowledge from itself without any guidance from extra networks. This paper proposes to perform Self-KD from image Mixture (MixSKD), which integrates these two techniques into a unified framework. MixSKD mutually distills feature maps and probability distributions between the random pair of original images and their mixup images in a meaningful way. Therefore, it guides the network to learn cross-image knowledge by modelling supervisory signals from mixup images. Moreover, we construct a self-teacher network by aggregating multi-stage feature maps for providing soft labels to supervise the backbone classifier, further improving the efficacy of self-boosting. Experiments on image classification and transfer learning to object detection and semantic segmentation demonstrate that MixSKD outperforms other state-of-the-art Self-KD and data augmentation methods.

Introduction

Knowledge Distillation (KD) typically involves a large "teacher" model transferring its learned knowledge to a smaller "student" model. While effective, this paradigm requires a computationally expensive two-stage training process. Self-Knowledge Distillation (Self-KD) addresses this by enabling a network to act as its own teacher. However, traditional Self-KD methods often focus on "intra-image" consistency (e.g., between different augmentations of the same image), potentially neglecting the "cross-image" relationships that define category boundaries.

MixSKD (Self-Knowledge Distillation from Mixup) bridges this gap by leveraging the Mixup data augmentation strategy. By blending two random images, the model is forced to learn linear relationships between categories in both its internal feature representations and its final predictions. This approach acts as a powerful regularizer, reducing undesirable oscillations and improving the model's ability to capture "dark knowledge" regarding inter-class similarities.

Methodology

The MixSKD framework integrates Mixup directly into the distillation process across three primary levels: feature maps, probability distributions, and an auxiliary self-teacher network.

Conceptual overview of MixSKD showing the relationship between mixed and original images at feature, probability, and label levels.

Feature-level Self-KD

The network is encouraged to maintain linearity in its feature space. Specifically, the feature map of a mixed image should ideally match the weighted sum of the feature maps of the two source images. To enhance this process, MixSKD employs an adversarial discriminator that distinguishes between "real" mixed features and interpolated ones, forcing the backbone to learn more robust semantic features.

Logit-level Self-KD

Similarly, the final logits (probability distributions) for the mixed image are distilled to match the weighted combination of the source images' distributions. Unlike standard Mixup, which uses hard labels, MixSKD uses the network's own "soft" predictions. These soft labels contain rich information about class similarities (e.g., a cat is more similar to a dog than a ship), providing a more nuanced supervisory signal.

The Self-Teacher Network

To further boost performance, the authors introduce a self-teacher network that aggregates feature maps from multiple stages of the backbone. This module provides high-quality soft labels to supervise the main classifier.

Detailed architecture of the MixSKD framework, illustrating multi-stage feature alignment and the self-teacher network.

Importantly, all auxiliary components—including the discriminators and the self-teacher network—are discarded after training. This ensures that the final model maintains the same inference speed as the baseline architecture.

Experimental Results

MixSKD was evaluated across diverse benchmarks, including CIFAR-100, ImageNet-1K, and several fine-grained classification datasets.

  • Classification Accuracy: On CIFAR-100, a ResNet-18 trained with MixSKD achieved 80.32% accuracy, significantly outperforming the 76.24% baseline and previous state-of-the-art Self-KD methods like FRSKD (77.71%). On ImageNet-1K, a ResNet-50 reached 78.76% top-1 accuracy, a 1.25% improvement over standard Mixup.
  • Adversarial Robustness: Under FGSM white-box attacks, MixSKD maintained 15.3% accuracy, compared to 8.6% for the baseline and 11.4% for standard Mixup, indicating the learning of more robust features.
  • Transfer Learning: When used as a backbone for object detection (COCO 2017) and semantic segmentation (Pascal VOC, ADE20K), MixSKD provided higher mAP and mIoU scores than competing KD methods, demonstrating the generalizability of its learned features.

Conclusion

MixSKD successfully unifies Mixup augmentation and Self-KD into a single framework that captures cross-image knowledge. By enforcing linearity across feature and logit spaces and utilizing a multi-stage self-teacher, it achieves state-of-the-art performance without increasing inference costs. Future work may explore integrating spatial-consistency augmentations like CutMix to further refine the localization capabilities of the distilled models.

Summary updated

Abstract

Unlike the conventional Knowledge Distillation (KD), Self-KD allows a network to learn knowledge from itself without any guidance from extra networks. This paper proposes to perform Self-KD from image Mixture (MixSKD), which integrates these two techniques into a unified framework. MixSKD mutually distills feature maps and probability distributions between the random pair of original images and their mixup images in a meaningful way. Therefore, it guides the network to learn cross-image knowledge by modelling supervisory signals from mixup images. Moreover, we construct a self-teacher network by aggregating multi-stage feature maps for providing soft labels to supervise the backbone classifier, further improving the efficacy of self-boosting. Experiments on image classification and transfer learning to object detection and semantic segmentation demonstrate that MixSKD outperforms other state-of-the-art Self-KD and data augmentation methods. The code is available at https://github.com/winycg/Self-KD-Lib.
BibTeX
@inproceedings{yang2022mixskd,
  author = {Chuanguang Yang and Zhulin An and Helong Zhou and Linhang Cai and Xiang Zhi and Jiwen Wu and Yongjun Xu and Qian Zhang},
  title = {{MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-20053-3\_31},
}
Figure