SSSD: Self-Supervised Self Distillation

Wei-Chi Chen, Wei-Ta ChuWACV 2023

TL;DR

This paper introduces Self-Supervised Self Distillation (SSSD), a novel approach that enables knowledge distillation without requiring any labeled data. By leveraging self-supervised learning to generate pseudo-labels, SSSD allows compact models to learn effectively, outperforming existing unsupervised methods.

What Problem Does This Paper Address?

Knowledge distillation (KD) is a powerful technique for transferring knowledge from large, complex "teacher" models to smaller, more efficient "student" models. Traditionally, KD relies on a pre-trained teacher model and, crucially, labeled data to guide the distillation process. However, obtaining large, labeled datasets can be expensive and time-consuming, and in some scenarios, such data may not be available at all. While "self-distillation" methods have emerged to eliminate the need for a separate teacher model (allowing a single network to act as both teacher and student), they still typically require labeled data. This paper addresses the gap of performing knowledge distillation in a completely unsupervised manner, where no labeled data is available for the target task.

Methods

The core of the proposed SSSD method involves two main stages:

  1. Self-Supervised Pre-training (SSL): A model (e.g., ResNet-18 or ResNet-50) is first trained on an unlabeled dataset ($D_{pre}$) using a self-supervised learning task, specifically instance discrimination (similar to MoCo v2). This pre-training phase equips the model with strong visual representation learning capabilities without needing any labels.
  2. Self-Supervised Self Distillation (SSSD):
    • Pseudo-label Generation: The pre-trained SSL model is then used to extract features from the target dataset ($D_{tar}$). These features are clustered using the K-means algorithm. Data points within the same cluster are assigned the same "pseudo-label."
    • Self Distillation: A student model (e.g., a ResNet architecture split into multiple sections) is trained using these generated pseudo-labels. The deepest section of the student model acts as the "teacher," distilling knowledge to the shallower "student" sections. The distillation process is guided by three loss functions:
  3. Cross-entropy loss: Between the student classifiers' outputs and the pseudo-labels.
  4. KL divergence loss: Between the teacher classifier's output and the student classifiers' outputs, encouraging similarity in predicted distributions.
  5. L2 loss: Between feature maps of the deepest section and shallower sections, promoting feature consistency.

The key novelty lies in using SSL to generate the pseudo-labels, thereby enabling self-distillation without any human-annotated labels for the target task.

Datasets

The paper evaluates SSSD on the following datasets:

  • CIFAR-10
  • CIFAR-100
  • ImageNet-1K

For SSL pre-training ($D_{pre}$), the authors use these datasets. For the self-distillation phase ($D_{tar}$), they also use these datasets. They explore scenarios where $D_{pre}$ and $D_{tar}$ are the same, and also where $D_{pre}$ is ImageNet-1K and $D_{tar}$ is CIFAR-10/100.

Key Insights and Ideas

  • SSL as a powerful pseudo-label generator: The success of SSSD hinges on the ability of SSL to learn meaningful representations that can be effectively clustered into useful pseudo-labels, even without explicit task-specific supervision.
  • Unsupervised KD is feasible: The paper demonstrates that knowledge distillation can be performed effectively without any labeled data, significantly expanding its applicability.
  • Pre-training dataset scale matters: Using a larger-scale dataset (ImageNet-1K) for SSL pre-training generally leads to better pseudo-labels and improved performance on downstream tasks (CIFAR-10/100) compared to using smaller datasets for pre-training.
  • More SSL training epochs yield better results: A well-trained SSL model, trained for more epochs, provides more accurate pseudo-labels and richer knowledge, leading to higher accuracy in the self-distillation process.

Key Results

  • Performance on CIFAR-10, CIFAR-100, and ImageNet-1K: SSSD consistently achieves competitive classification accuracies, outperforming random labeling baselines by a significant margin. For instance, on ImageNet-1K, SSSD with a ResNet-50 SSL model achieves an ensemble accuracy of 61.12%, compared to 16.24% for random labels.
  • Comparison with other methods: SSSD slightly outperforms existing methods like SEED [7] and ClusterFit [23] on ImageNet-1K, demonstrating its effectiveness in unsupervised distillation.
  • Object Detection Transfer: Representations learned by SSSD can be transferred to other tasks. When used as a backbone for Faster R-CNN, SSSD shows superior performance compared to SEED.
  • Semi-Supervised Learning: SSSD-learned representations are robust, allowing for good performance even with very limited labeled data (e.g., 10% of ImageNet-1K) for fine-tuning.

Limitations and Future Work

  • Pseudo-label quality: While effective, the quality of pseudo-labels is still dependent on the clustering algorithm and the quality of features extracted by the SSL model. The paper notes that the performance of SSSD is influenced by the number of clusters used.
  • Dataset mismatch: The paper explores $D_{pre} \neq D_{tar}$ but primarily focuses on scenarios where $D_{pre}$ is ImageNet-1K. Further investigation into optimal dataset pairings could be beneficial.
  • Future Work: The authors suggest that their work breaks the limitation of well-labeled data for self-distillation, opening avenues for more research in unsupervised learning and its applications. The effectiveness of SSSD in other downstream tasks beyond classification and object detection could be explored.

Summary updated

BibTeX
@inproceedings{chen2023sssd,
  author = {Wei-Chi Chen and Wei-Ta Chu},
  title = {{SSSD: Self-Supervised Self Distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure