Introduction
Self-supervised representation learning, particularly contrastive learning, has become a cornerstone of modern computer vision. By using "instance discrimination"—treating every image as its own unique class—models can learn powerful features without human labels. However, this paradigm faces a significant hurdle: semantic collapse in lightweight models.
While high-capacity models like ResNet-50 can successfully group semantically similar images (e.g., different dogs) despite being told they are distinct, small models like AlexNet or MobileNet lack the capacity to do so. In these lightweight architectures, the feature space becomes fragmented, and the gap between unsupervised and supervised performance remains vast.
The ReKD Framework
To address this, we propose Relation Knowledge Distillation (ReKD). Instead of forcing a small "student" model to learn in isolation, ReKD introduces a larger "teacher" model to guide the learning process by identifying semantic relationships between images.
The framework consists of three core components:
- Online Heterogeneous Teacher: A high-capacity model (e.g., ResNet-50) is trained alongside the student. This "online" approach ensures the teacher provides a dynamic curriculum that evolves with the student's progress.
- Relation Miner: This module maintains a bank of semantic prototypes. It assigns images to these prototypes based on the teacher's features. If two images are assigned to the same prototype, they are considered a "positive pair."
- Relation Contrastive Loss: Unlike standard contrastive learning that uses a single positive (the image itself), ReKD employs a multi-positive objective. The student is encouraged to pull its features closer to all semantically related candidates identified by the teacher.
Experimental Results
We evaluated ReKD on ImageNet using the standard linear evaluation protocol across various lightweight backbones. The results demonstrate that ReKD effectively bridges the performance gap:
- AlexNet: ReKD achieved 50.1% top-1 accuracy, nearly matching the 50.5% achieved by fully supervised training. This represents a breakthrough for unsupervised learning on this architecture.
- EfficientNet-b0: Performance soared to 63.4%, a massive improvement over the 38.6% baseline from MoCov2.
- MobileNet-V3: Accuracy increased from 35.3% to 56.7%.
These gains highlight that the primary bottleneck for small models in self-supervised learning is not their architecture per se, but the rigid "one-positive" constraint of traditional contrastive objectives.
Conclusion
Relation Knowledge Distillation (ReKD) provides a path forward for training efficient, lightweight models without labels. By leveraging the relational knowledge of a larger teacher, small models can overcome semantic collapse and learn feature spaces that rival those produced by supervised learning. While the method requires additional compute during training due to the teacher model, the resulting lightweight backbones are highly effective for deployment on edge devices.
Summary updated