Rethinking Open-World Semi-Supervised Learning: Distribution Mismatch and Inductive Inference

Seongheon Park, Hyuk Kwon, Kwanghoon Sohn, Kibok LeearXiv 2024 · DOI · arXiv

Introduction

Open-World Semi-Supervised Learning (OWSSL) represents a significant advancement in machine learning, aiming to train models on a mixture of labeled and unlabeled data where the unlabeled set contains both known classes and entirely novel categories. However, current OWSSL research often operates under idealized conditions that do not reflect the complexities of "in-the-wild" data. This paper introduces Realistic Open-World Semi-Supervised Learning (ROWSSL), a framework designed to address these discrepancies by focusing on long-tailed distributions and inductive generalization.

The Problem: Unrealistic Assumptions in OWSSL

The authors argue that existing OWSSL methods rely on two primary "crutches" that limit their real-world applicability:

  1. Balanced Distributions: Most current benchmarks assume that labeled and unlabeled data share a perfectly balanced class distribution. In reality, most datasets are long-tailed, where a few "head" classes contain the majority of samples, while many "tail" classes are sparsely represented.
  2. Transductive Bias: Evaluation typically focuses on transductive learning—how well a model categorizes the specific unlabeled training data it has already seen. In practical applications, such as medical diagnostics, models must perform inductive inference, identifying novel classes in completely new, unseen patients.
Figure 1: The ROWSSL framework addresses distribution mismatches (matched vs. mismatched priors) and prioritizes inductive inference over transductive evaluation.

Proposed Method: Density-based Temperature scaling and Soft pseudo-labeling (DTS)

To overcome these challenges, the authors propose DTS, a method that leverages local data density to handle class imbalances and improve generalization. The core of DTS consists of three innovative components:

Tailedness Estimation

Since true class priors are unknown in unlabeled data, DTS uses "tailedness" as a proxy. The method assumes that samples from rare "tail" classes will be more sparsely distributed in the feature space. By maintaining a queue of features and using $k$-means to establish "tailedness prototypes," the model can estimate the local density of every sample without needing explicit labels.

Figure 2: Visualization of Tailedness Prototypes and Momentum Embeddings used to estimate local density for head and tail classes.

Dynamic Temperature Scaling

In contrastive learning, the temperature parameter ($\tau$) dictates the model's focus on sample difficulty. DTS dynamically assigns a smaller $\tau$ to tail classes to encourage the learning of instance-specific features, while a larger $\tau$ is assigned to head classes to preserve their broader semantic structure. This prevents the model from being overwhelmed by the high frequency of head class samples.

Uncertainty-Aware Soft Pseudo-labeling

Models trained on long-tailed data are naturally biased toward head classes. DTS calculates "class uncertainty" based on the variance of tailedness scores within a class. This uncertainty is used to adjust the classifier's output, preventing the model from over-confidently misclassifying novel samples into well-known head categories.

Experimental Evaluation

The authors evaluated DTS on three major benchmarks modified for long-tailed, open-world scenarios: CIFAR-100-LT, Herbarium19 (a naturally long-tailed plant dataset), and ImageNet-100-LT. They tested scenarios where labeled and unlabeled distributions were both matched (MCAR) and mismatched (MNAR).

Key Results

  • Superior Accuracy: On CIFAR-100-LT, DTS achieved a transductive accuracy of 65.3%, significantly outperforming previous state-of-the-art methods like BaCon (45.8%) and SimGCD (51.7%).
  • Novel Class Discovery: On the Herbarium19 dataset, DTS showed a massive improvement in discovering novel classes, reaching 46.8% accuracy compared to SimGCD's 35.8%.
  • Inductive Generalization: Unlike many existing methods whose performance collapses when forced to classify unseen samples one at a time, DTS maintained high performance across inductive inference tests.

Conclusion and Future Work

DTS demonstrates that local density is a powerful signal for adjusting learning dynamics in the absence of class priors. By rethinking the geometry of the feature space through temperature scaling, the model achieves better balance between head and tail classes. While the method assumes labeled and unlabeled data come from similar sources, future work could explore making these density-based proxies robust to significant domain shifts or extreme data noise.

Summary updated

Abstract

Open-world semi-supervised learning (OWSSL) extends conventional semi-supervised learning to open-world scenarios by taking account of novel categories in unlabeled datasets. Despite the recent advancements in OWSSL, the success often relies on the assumptions that 1) labeled and unlabeled datasets share the same balanced class prior distribution, which does not generally hold in real-world applications, and 2) unlabeled training datasets are utilized for evaluation, where such transductive inference might not adequately address challenges in the wild. In this paper, we aim to generalize OWSSL by addressing them. Our work suggests that practical OWSSL may require different training settings, evaluation methods, and learning strategies compared to those prevalent in the existing literature.
BibTeX
@misc{park2024rethinking,
  author = {Seongheon Park and Hyuk Kwon and Kwanghoon Sohn and Kibok Lee},
  title = {{Rethinking Open-World Semi-Supervised Learning: Distribution Mismatch and Inductive Inference}},
  year = {2024},
  eprint = {2405.20829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.20829},
}
Figure