Semi-Supervised Domain Generalization with Known and Unknown Classes

Lei Zhang, Ji-Fu Li, Wei Wang

Introduction

Most semi-supervised learning methods operate under a closed-world assumption: they presume we know every category the model will ever encounter. However, real-world data is rarely so tidy. This paper introduces CWAEE (Class-Wise Adaptive Exploration and Exploitation), a novel framework designed to handle the messy reality where unlabeled training data contains unknown classes. By leveraging adaptive thresholds and Fourier-based data augmentation, CWAEE successfully generalizes known classes to entirely new domains while accurately flagging unknown ones.

The Open-Set Domain Generalization Problem

In traditional machine learning, models rely on the "i.i.d. assumption"—that training and testing data are drawn from the exact same distribution. Domain Generalization (DG) challenges this by requiring a model to learn from several distinct "source" domains (e.g., photos and cartoons) and perform accurately on a completely unseen "target" domain (e.g., sketches).

Semi-Supervised Domain Generalization (SSDG) makes this setup more practical by utilizing a small set of labeled samples alongside a massive pool of unlabeled data. Unfortunately, existing SSDG methods have a critical blind spot: they assume every unlabeled image belongs to one of the "known" classes present in the labeled set.

The Open-Set SSDG Problem Setting

In realistic scenarios, unlabeled data is "open-set." It contains "unknown" classes that the model has never been trained to recognize. If a model is forced to assign a known label (like "dog") to an unlabeled image of an unknown class (like "guitar"), it suffers from negative transfer, severely degrading its performance on actual known classes.

The CWAEE Methodology

To tackle this open-set challenge, the authors propose CWAEE, which divides the learning process into two distinct phases: Exploration and Exploitation.

1. Exploration (Detecting Unknowns)

  • One-vs-Rest Classifiers: Instead of relying on a standard "winner-takes-all" softmax classifier—which is notoriously overconfident on out-of-distribution data—CWAEE employs one-vs-rest classifiers using sigmoid functions. This structural change allows the model to effectively say "I don't know" when an image doesn't fit any known category.
  • Adaptive Thresholds via BMM: The framework abandons fixed confidence thresholds (e.g., a rigid 0.9 cutoff). Instead, it uses a Beta Mixture Model (BMM) to dynamically learn class-specific thresholds. This adapts to the reality that some classes are inherently harder to identify than others, allowing the model to decide what "confident" looks like for each specific category.

2. Exploitation (Improving Generalization)

  • Fourier Transformation for Style Swapping: To build robustness against domain shifts, CWAEE utilizes a clever Fourier transform technique. It swaps the "amplitude" spectrum (which encodes the "style" or domain characteristics of an image, like brushstrokes or lighting) between different samples while preserving the "phase" spectrum (which retains the "semantics" or actual object shape).
  • Consistency Regularization: The model is then trained to output the same prediction for an original image and its style-swapped counterpart. This consistency forces the network to ignore superficial domain noise and focus on high-level, generalizable semantic features.

Datasets and Evaluation

The researchers evaluated CWAEE on three standard computer vision benchmarks, specifically modified to include "seen unknown" (in unlabeled source data) and "unseen unknown" (in target data) classes:

  • PACS: A diverse dataset containing Photos, Art, Cartoons, and Sketches.
  • Office-Home: A complex dataset with 65 classes across Art, Clipart, Product, and Real-World domains.
  • miniDomainNet: A large-scale subset of DomainNet featuring 126 classes.

Key Results

CWAEE consistently outperformed state-of-the-art methods, such as FixMatch and StyleMatch, across all evaluated datasets.

Performance across varying numbers of unknown classes
  • Accuracy on Known Classes: On the PACS dataset, CWAEE achieved an average accuracy of 84.20%, significantly outpacing the next best competitor (StyleMatch at 80.69%).
  • Detecting Unknowns (AUROC): The method demonstrated a massive leap in its ability to identify unknown classes. On PACS, it reached an AUROC of 79.21%, whereas most baseline methods struggled between 50% and 60%.
  • Robustness to Unknowns: As shown in the results, CWAEE maintains high accuracy and AUROC even as the number of "seen unknown" classes in the unlabeled data increases, proving its resilience in highly noisy environments.

Limitations and Future Directions

While powerful, CWAEE has a few limitations. It relies on a small validation set of known classes to calibrate temperature scaling for the classifiers—data that might be unavailable in the strictest domain generalization settings. Additionally, maintaining score queues and fitting Beta Mixture Models for every class introduces computational overhead compared to simpler SSDG approaches.

Looking forward, the authors suggest that this adaptive exploration logic could be highly valuable in safety-critical fields like medical diagnosis or autonomous driving, where accurately identifying "unknown" scenarios is just as important as classifying known ones.

Summary updated

Abstract

Semi-Supervised Domain Generalization (SSDG) aims to learn a model that is generalizable to an unseen target domain with only a few labels, and most existing SSDG methods assume that unlabeled training and testing samples are all known classes. However, a more realistic scenario is that known classes may be mixed with some unknown classes in unlabeled training and testing data. To deal with such a scenario, we propose the Class-Wise Adaptive Exploration and Exploitation (CWAEE) method. In particular, we explore unlabeled training data by using one-vs-rest classifiers and class-wise adaptive thresholds to detect known and unknown classes, and exploit them by adopting consistency regularization on augmented samples based on Fourier Transformation to improve the unseen domain generalization. The experiments conducted on real-world datasets verify the effectiveness and superiority of our method.
BibTeX
@inproceedings{zhang2023semi,
  author = {Lei Zhang and Ji-Fu Li and Wei Wang},
  title = {{Semi-Supervised Domain Generalization with Known and Unknown Classes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=ce59j806df},
}
Figure