The Others: Naturally Isolating Out-of-Distribution Samples for Robust Open-Set Semi-Supervised Learning

You Rim Choi, Subeom Park, Seojun Heo, Eunchung Noh, Hyung-Sin KimarXiv 2025 · DOI · arXiv

Introduction

In standard Semi-Supervised Learning (SSL), models typically assume that all unlabeled data belongs to a set of known "In-Distribution" (ID) classes. However, real-world unlabeled pools are often contaminated with "Out-of-Distribution" (OOD) samples from classes the model has never encountered. This scenario, known as Open-Set Semi-Supervised Learning (OSSL), presents a significant challenge: how to leverage unlabeled data without being misled by novel OOD samples.

Existing OSSL methods often fall into two extremes. Aggressive approaches attempt to force-align OOD samples into a synthetic "catch-all" class, which often leads to "geometric collapse"—where the model overfits to specific OOD examples and fails to generalize to unseen OOD types. Conversely, conservative methods simply discard low-confidence samples, losing valuable geometric cues that could help define the boundaries of the ID space. SkipAlign proposes a middle ground through a principle called Selective Non-Alignment (SNA).

Methodology: Selective Non-Alignment

The core of the SkipAlign framework is treating the embedding space like a universe of compact ID "galaxies" surrounded by a vast OOD "interstellar void." While traditional contrastive learning relies on "Pulling" (attracting similar samples) and "Pushing" (repelling dissimilar ones), SNA introduces a third operator: Skip.

Overview of the SkipAlign framework and the Selective Non-Alignment (SNA) concept.

Dual-Gate ID Selection

To determine which samples to "pull," SkipAlign employs a Dual-Gate selection mechanism. A sample must pass high-confidence thresholds from both a standard ID classifier and a specialized One-vs-All (OVA) OOD detector. Only these "confident ID" samples are pulled toward class prototypes.

The "Skip" Operator and Angular Updates

For samples that fail the dual-gate check—including both seen OOD and uncertain ID samples—the model skips the pull operation. Instead, it applies a non-selective angular update. This provides a "torque" that rotates uncertain features away from ID prototypes without pinning them to a specific synthetic class. This prevents geometric collapse and allows the OOD space to remain naturally heterogeneous.

Key Insights: Norm as a Signal

A striking consequence of the SNA strategy is the emergence of the feature norm as a powerful indicator for OOD detection. Because the model only "pulls" (and thus increases the feature norm of) confident ID samples, these samples naturally form dense, high-norm clusters. In contrast, OOD samples remain dispersed in a low-norm, sparse region of the feature space.

Visualization of feature space and norm distribution, demonstrating the isolation of OOD samples.

As shown in the empirical visualizations above, ID classes (like Bird, Cat, Dog) exhibit significantly higher feature norms compared to both seen and unseen OOD samples. This separation allows the model to naturally spot anything that does not resemble a known "galaxy."

Experimental Results

SkipAlign was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and ImageNet-30, specifically testing for unseen OOD generalization. The method consistently outperformed state-of-the-art baselines like SCOMatch and ProSub:

  • OOD Detection: SkipAlign improved the Overall-OOD AUC by an average of 3.1 points, with gains reaching up to 7.1 points in difficult scenarios.
  • ID Accuracy: These gains in OOD detection did not compromise classification performance; the model maintained or slightly improved closed-set accuracy across datasets.
  • Robustness: The framework proved equally effective at detecting "seen" OOD (classes present in the unlabeled training pool) and "unseen" OOD (entirely new classes encountered only at test time).

Conclusion and Future Work

SkipAlign demonstrates that by "letting the void be void," models can achieve superior robustness in open-set environments. By avoiding the forced alignment of uncertain samples, the framework preserves the natural geometry of the feature space. Future research may focus on improving detection for OOD classes that have high semantic overlap with ID classes and further exploring how different network layers contribute to the SNA signal.

Summary updated

Abstract

Open-set semi-supervised learning (OSSL) leverages unlabeled data containing both in-distribution (ID) and unknown out-of-distribution (OOD) samples, aiming simultaneously to improve closed-set accuracy and detect novel OOD instances. Existing methods either discard valuable information from uncertain samples or force-align every unlabeled sample into one or a few synthetic "catch-all" representations, resulting in geometric collapse and overconfidence on only seen OODs. To address the limitations, we introduce selective non-alignment, adding a novel "skip" operator into conventional pull and push operations of contrastive learning. Our framework, SkipAlign, selectively skips alignment (pulling) for low-confidence unlabeled samples, retaining only gentle repulsion against ID prototypes. This approach transforms uncertain samples into a pure repulsion signal, resulting in tighter ID clusters and naturally dispersed OOD features. Extensive experiments demonstrate that SkipAlign significantly outperforms state-of-the-art methods in detecting unseen OOD data without sacrificing ID classification accuracy.
BibTeX
@misc{choi2025others,
  author = {You Rim Choi and Subeom Park and Seojun Heo and Eunchung Noh and Hyung-Sin Kim},
  title = {{The Others: Naturally Isolating Out-of-Distribution Samples for Robust Open-Set Semi-Supervised Learning}},
  year = {2025},
  eprint = {2504.12569},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.12569},
}
Figure