Provably Uncertainty-Guided Universal Domain Adaptation

Yifan Wang, Lin Zhang, Ran Song, Lin Ma, Wěi Zhāng, Zhang, WeiarXiv 2022 · DOI · arXiv

Abstract

Universal domain adaptation (UniDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain without prior assumptions about their label sets. This requires accurately distinguishing unknown samples from known ones in the target domain. A primary challenge in UniDA is domain misalignment caused by nonidentical label sets, which often leads models to be biased toward common classes and produce overconfident predictions for unknown samples. To address these issues, we propose a new uncertainty-guided UniDA framework. We introduce an empirical estimation of the probability that a target sample belongs to an unknown class by exploiting its distribution in the latent space. Based on this, we develop a neighbors searching scheme in a linear subspace with a $\delta$-filter to estimate uncertainty and discover unknown samples. Additionally, we implement an uncertainty-guided margin loss to balance prediction confidences for both known and unknown samples. Experiments on three public datasets demonstrate that our method significantly outperforms existing state-of-the-art approaches.

Introduction: The Challenge of Universal Domain Adaptation

In traditional Domain Adaptation (DA), the source and target domains are assumed to share the same label sets. Universal Domain Adaptation (UniDA) relaxes this assumption, allowing for "private" classes in either domain and "unknown" classes in the target domain that were never seen during source training.

Domain overlap and feature space comparison between existing methods and the proposed approach.

As illustrated in the figure above, two major hurdles persist in UniDA:

  1. Domain Misalignment: Distribution shifts can cause "unknown" target samples to appear similar to "known" source samples in the feature space.
  2. Model Overconfidence: Supervised training on the source domain biases the model, forcing it to categorize target samples into known classes with high confidence, even when they belong to unknown categories.

The Proposed Framework

To mitigate these challenges, we propose a three-pronged strategy that focuses on neighborhood consistency and uncertainty-guided learning.

Overview of the uncertainty-guided UniDA framework architecture.

1. Neighbors Searching in Linear Subspace (NSLS)

Rather than analyzing raw features, our model projects data into a lower-dimensional linear subspace using Singular Value Decomposition (SVD). This projection acts as a denoising tool, making the data distribution more uniform and making it easier to identify outliers (unknowns) that might otherwise be hidden near the boundaries of known clusters.

2. The $\delta$-filter for Outlier Detection

To distinguish between a "known" sample and a "noisy unknown," we introduce the $\delta$-filter. This mechanism compares the compactness of a target sample’s neighborhood against a randomly sampled source neighborhood. By measuring the change in local variance (via the eigenvalues of the covariance matrix) when a target sample is added, the filter can reliably flag samples that disrupt the local distribution as "unknown."

3. Uncertainty-Guided Margin Loss (UGM)

Standard cross-entropy losses often result in overly tight representations for source classes. Our UGM loss introduces a dynamic margin during training, adjusted based on the uncertainty of discovered unknown samples. This effectively "leaves room" in the feature space for the unknown class, balancing the intra-class variance and reducing the gap between known and unknown class confidences.

Key Insights and Results

Our framework was evaluated on three major benchmarks: Office-31, OfficeHome, and VisDA. The results consistently showed that our method outperforms state-of-the-art (SOTA) models like OVANet and GATE.

  • Performance Gains: On the VisDA dataset, we achieved a 3.7% improvement in the Open-Set DA (ODA) setting. On OfficeHome, we saw a significant jump of 5.3% in the same setting.
  • Ablation Significance: The importance of the UGM loss was particularly evident in ablation studies; removing it caused the H-score (the harmonic mean of accuracy on known and unknown classes) to drop from 87.3 to 29.2 in specific tasks.
  • Theoretical Reliability: Crucially, we provide a theoretical foundation proving that our empirical estimation of the "unknown" probability is mathematically sound, ensuring the framework's robustness across different data distributions.

Conclusion

By prioritizing neighborhood compactness over simple proximity and utilizing uncertainty to guide the learning margin, our framework provides a robust solution to the overconfidence and misalignment problems in UniDA. While the method introduces some computational overhead due to SVD and memory bank maintenance, its superior ability to discover unknown samples sets a new benchmark for universal domain adaptation tasks.

Summary updated

Abstract

Universal domain adaptation (UniDA) aims to transfer the knowledge from a labeled source domain to an unlabeled target domain without any assumptions of the label sets, which requires distinguishing the unknown samples from the known ones in the target domain. A main challenge of UniDA is that the nonidentical label sets cause the misalignment between the two domains. Moreover, the domain discrepancy and the supervised objectives in the source domain easily lead the whole model to be biased towards the common classes and produce overconfident predictions for unknown samples. To address the above challenging problems, we propose a new uncertainty-guided UniDA framework. Firstly, we introduce an empirical estimation of the probability of a target sample belonging to the unknown class which fully exploits the distribution of the target samples in the latent space. Then, based on the estimation, we propose a novel neighbors searching scheme in a linear subspace with a $δ$-filter to estimate the uncertainty score of a target sample and discover unknown samples. It fully utilizes the relationship between a target sample and its neighbors in the source domain to avoid the influence of domain misalignment. Secondly, this paper well balances the confidences of predictions for both known and unknown samples through an uncertainty-guided margin loss based on the confidences of discovered unknown samples, which can reduce the gap between the intra-class variances of known classes with respect to the unknown class. Finally, experiments on three public datasets demonstrate that our method significantly outperforms existing state-of-the-art methods.
BibTeX
@misc{wang2022provably,
  author = {Yifan Wang and Lin Zhang and Ran Song and Lin Ma and Wěi Zhāng and Zhang, Wei},
  title = {{Provably Uncertainty-Guided Universal Domain Adaptation}},
  year = {2022},
  eprint = {2209.09616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.09616},
}
Figure