Safe Semi-Supervised Contrastive Learning Using In-Distribution Data as Positive Examples

Mingu Kwak, Hyungu Kahng, Seoung Bum Kim

Abstract

Semi-supervised learning methods have shown promising results in solving many practical problems when only a few labels are available. The existing methods assume that the class distributions of labeled and unlabeled data are equal; however, their performances are significantly degraded in class distribution mismatch scenarios where out-of-distribution (OOD) data exist in the unlabeled data. Previous safe semi-supervised learning studies have addressed this problem by making OOD data less likely to affect training based on labeled data. However, even if the studies effectively filter out the unnecessary OOD data, they can lose the basic information that all data share regardless of class. To this end, we propose to apply a self-supervised contrastive learning approach to fully exploit a large amount of unlabeled data. We also propose a contrastive loss function with coefficient schedule to aggregate as an anchor the labeled negative examples of the same class into positive examples. To evaluate the performance of the proposed method, we conduct experiments on image classification datasets - CIFAR-10, CIFAR-100, Tiny ImageNet, and CIFAR-100+Tiny ImageNet - under various mismatch ratios. The results show that self-supervised contrastive learning significantly improves classification accuracy. Moreover, aggregating the in-distribution examples produces better representation and consequently further improves classification accuracy.

Introduction: The Challenge of Class Distribution Mismatch

Standard Semi-Supervised Learning (SSL) algorithms typically operate under a "closed-world" assumption, where the small labeled dataset and the large unlabeled dataset are assumed to share the exact same classes. However, in real-world applications, unlabeled data is often collected from diverse sources and inevitably contains "noise" or out-of-distribution (OOD) data—classes that were never present in the labeled set. For example, a dog-vs-cat classifier trained with unlabeled web-scraped data might encounter images of cars, flowers, or birds.

Comparison between conventional SSL and the realistic scenario involving OOD data.

When standard SSL models encounter this class distribution mismatch, their performance often degrades significantly. Previous "safe SSL" attempts tried to mitigate this by filtering out OOD data. This research argues that such filtering is counterproductive: even OOD images share low-level visual features (textures, lighting, edges) with in-distribution (ID) data. By discarding OOD data, models lose valuable information that could improve the backbone encoder's general feature representation.

Proposed Method: Safe Semi-Supervised Contrastive Learning (SSCL)

The proposed framework, Safe Semi-Supervised Contrastive Learning (SSCL), is built upon Momentum Contrast (MoCo), a popular self-supervised contrastive learning framework. SSCL introduces two primary innovations to handle messy unlabeled data effectively:

1. ID-aware Contrastive Loss ($L^{ID}$)

In standard instance discrimination, every image is treated as its own unique class. SSCL modifies this by leveraging available labels. When the model processes a labeled "dog" (the anchor), it searches its memory queue for other labeled "dogs." Instead of pushing these same-class samples away as negatives, the model "pulls" them closer as additional positives. This mechanism forces the model to cluster ID classes more tightly during the pre-training phase.

The SSCL architecture showing the memory queue and the "PULL" mechanism for labeled in-distribution data.

2. The Coefficient Schedule ($w(t)$)

Using labeled data too aggressively during pre-training can lead to overfitting, particularly when labels are scarce. To prevent this, the authors introduced a linear decay schedule for the ID-aware loss. Early in training, the $L^{ID}$ loss is strong to help define initial class boundaries. As training progresses, its influence fades, allowing the model to focus on learning the broader data manifold through standard instance discrimination across the entire unlabeled set (including OOD data).

Experimental Results

The method was evaluated across several benchmarks including CIFAR-10, CIFAR-100, and Tiny ImageNet, simulating mismatch ratios from 0% to 100% (where the unlabeled data contains entirely different classes than the labeled set).

  • Superior Accuracy: In a CIFAR-10 scenario with a 50% mismatch ratio, SSCL achieved 80.64% accuracy, outperforming the MoCo baseline (80.10%) and significantly beating specialized safe SSL methods like TOOR (78.67%) and UASD (78.12%).
  • Robustness to High Mismatch: Even at a 100% mismatch ratio, the method maintained high performance, proving it can extract useful features from completely unrelated images without being "distracted" by the lack of class overlap.
  • Fine-Grained Success: On Tiny ImageNet, the performance gap widened, with SSCL outperforming the baseline by nearly 3%, suggesting the approach is particularly effective for complex, high-dimensional data.

Conclusion

This research demonstrates that OOD data in unlabeled sets should be viewed as a resource rather than just noise. By using a contrastive approach, models can learn general visual representations from all available data while using a decaying ID-aware loss to steer target class clusters. This "safe" framework provides a robust path forward for semi-supervised learning in unpredictable, real-world data environments.

Summary updated

Abstract

Semi-supervised learning methods have shown promising results in solving many practical problems when only a few labels are available. The existing methods assume that the class distributions of labeled and unlabeled data are equal; however, their performances are significantly degraded in class distribution mismatch scenarios where out-of-distribution (OOD) data exist in the unlabeled data. Previous safe semi-supervised learning studies have addressed this problem by making OOD data less likely to affect training based on labeled data. However, even if the studies effectively filter out the unnecessary OOD data, they can lose the basic information that all data share regardless of class. To this end, we propose to apply a self-supervised contrastive learning approach to fully exploit a large amount of unlabeled data. We also propose a contrastive loss function with coefficient schedule to aggregate as an anchor the labeled negative examples of the same class into positive examples. To evaluate the performance of the proposed method, we conduct experiments on image classification datasets - CIFAR-10, CIFAR-100, Tiny ImageNet, and CIFAR-100+Tiny ImageNet - under various mismatch ratios. The results show that self-supervised contrastive learning significantly improves classification accuracy. Moreover, aggregating the in-distribution examples produces better representation and consequently further improves classification accuracy.
BibTeX
@inproceedings{kwak2025safe,
  author = {Mingu Kwak and Hyungu Kahng and Seoung Bum Kim},
  title = {{Safe Semi-Supervised Contrastive Learning Using In-Distribution Data as Positive Examples}},
  booktitle = {IEEE Access},
  year = {2025},
  url = {https://openreview.net/forum?id=JPjKJIMF6x},
}
Figure