TL;DR
This paper introduces Open-Set Single Domain Generalization (OS-SDG), a realistic yet difficult machine learning task where a model trained on only one source domain must generalize to unseen domains containing entirely new "unknown" categories. To solve this, the authors propose CrossMatch, a framework that synthesizes "fake" unknown samples via adversarial augmentation and uses a novel consistency regularization between different classifier types to sharpen the model's ability to flag novel objects in the wild.
The Challenge of Open-Set Single Domain Generalization
Most research in Domain Generalization (DG) assumes a "closed-set" scenario: even if the environment changes (e.g., moving from photos to sketches), the list of possible categories remains the same. Single Domain Generalization (SDG) makes this harder by providing only one training domain.
However, real-world applications are rarely closed-set. An autonomous vehicle trained on city streets (source) might encounter a "snowmobile" or a "horse-drawn carriage" in a rural area (target)—categories it never saw during training. Existing SDG methods fail here; they try to force these "unknown" objects into "known" categories, leading to dangerous misclassifications.
The authors tackle the OS-SDG challenge: learning to classify known classes accurately while effectively identifying and rejecting samples from unknown classes, all while seeing only one source domain.
The CrossMatch Framework
To address OS-SDG, the authors propose CrossMatch, which operates in two alternating stages:
- Auxiliary Sample Generation (Maximization Stage): Since the model never sees "unknown" classes during training, CrossMatch creates them. It uses adversarial data augmentation to generate auxiliary samples ($x_u$) that are intentionally pushed away from the known class clusters in the feature space. It leverages a multi-binary classifier (a set of one-vs-all classifiers) to find regions where all classifiers agree that a sample does not belong to any known class.
- Cross-Classifier Consistency Regularization (CCR) (Minimization Stage): To make the main model better at identifying these "unknowns," the authors introduce a consistency loss. They take the output of the multi-binary classifier and the standard multi-class classifier and force them to produce similar "one-vs-all" probability distributions for the generated auxiliary samples. This "propagates" the knowledge of what an unknown sample looks like from the binary classifiers to the main feature extractor.
By training on both real source data and these synthesized "unknowns," the model learns a more robust decision boundary that includes a "rejection" zone for novel objects.
Key Insights
- Entropy is a double-edged sword: While high entropy (uncertainty) in a model's output is a common signal for "unknown" samples, standard models aren't trained to produce high entropy for novel classes specifically. CrossMatch explicitly trains the model to associate high entropy with the "out-of-source" regions it synthesizes.
- Synthesizing the "Unknown": The core insight is that we don't need real unknown data to learn a rejection boundary. By mathematically "inventing" samples that the current model finds confusing and then teaching the model to label them as unknown, we can preemptively prepare for real-world novelties.
- Architectural Synergy: Using two different classifier heads (multi-class vs. multi-binary) and forcing them to agree (CCR) acts as a powerful regularizer. The multi-binary head is naturally better at saying "none of the above," and CCR transfers this "skepticism" to the main classification head.
Datasets and Results
The authors evaluated CrossMatch on four standard benchmarks adapted for the open-set setting: Digits, Office31, Office-Home, and PACS. They used the h-score (the harmonic mean of known class accuracy and unknown class accuracy) as their primary metric, as it prevents models from performing well by simply calling everything "known" or everything "unknown."
- Significant Gains: On the complex Office-Home dataset, CrossMatch improved the h-score of standard SDG methods (like MEADA) by 13.13%.
- Robustness: On the Digits benchmark, CrossMatch + MEADA outperformed the baseline by over 8% in overall accuracy and nearly 9% in h-score.
- Better Rejection: Compared to OpenMax (a standard open-set recognition tool), CrossMatch was far more effective at handling domain shift, whereas OpenMax often struggled when the target domain looked visually different from the source.
Limitations and Future Work
- The Threshold Problem: During inference, the model relies on an entropy threshold ($\mu$) to decide if a sample is unknown. Choosing this threshold is difficult because, in a true SDG scenario, you have no target data to "tune" it on.
- Synthetic Gap: While the auxiliary samples help, there is still a gap between "mathematically synthesized unknowns" and "real-world novel objects." Future work could explore more sophisticated generative models (like GANs or Diffusion models) to create more realistic auxiliary samples.
- Single Source Bias: The model is still heavily biased toward the single source domain. If the source domain is very narrow, the "unknown" space might be too large to map effectively.
Summary updated