Unveiling the Unknown: Unleashing the Power of Unknown to Known in Open-Set Source-Free Domain Adaptation

Fuli Wan, Han Zhao, Xu Yang, Cheng DengCVPR 2024 · OpenReview

Abstract

Open-Set Source-Free Domain Adaptation aims to transfer knowledge in realistic scenarios where the target domain has additional unknown classes compared to the limited-access source domain. Due to the absence of information on unknown classes, existing methods mainly transfer knowledge of known classes while roughly grouping unknown classes as one, attenuating the knowledge transfer and generalization. In contrast, this paper advocates that exploring unknown classes can better identify known ones, and proposes a domain adaptation model to transfer knowledge on known and unknown classes jointly. Specifically, given a source pre-trained model, we first introduce an unknown diffuser that can determine whether classes in space need to be split and merged through similarity measures, to estimate and generate a wider class space distribution, including known and unknown classes. Based on such a wider space distribution, we enhance the reliability of known class knowledge in the source pre-trained model through contrastive constraint. Finally, various supervision information, including reliable known class knowledge and clustered pseudo-labels, optimize the model for impressive knowledge transfer and generalization. Extensive experiments show that our network can achieve superior exploration and knowledge generalization on unknown classes, while with excellent known class transfer.

Introduction

In real-world machine learning applications, models often encounter data from a "target" domain that differs significantly from their original "source" training data. This challenge is intensified in Open-Set Source-Free Domain Adaptation (OS-SFDA) scenarios, where two major constraints exist: first, the original source training data is inaccessible due to privacy or efficiency concerns (Source-Free); and second, the target domain contains "unknown" classes never seen during the source training phase (Open-Set).

Existing OS-SFDA methods typically attempt to force all unknown target data into a single "sink" class. However, this oversimplification often prevents the model from learning sharp decision boundaries, effectively "muddying the water" for the known classes it is intended to recognize. This paper proposes a paradigm shift: by actively discovering the underlying structure of the unknown space, we can better distinguish known classes from unknown ones.

Comparison of existing methods that group unknowns into a single class versus the proposed method that discovers individual unknown classes.

Methodology

The proposed framework, UPUK (Unknown to Known), introduces an Unknown Diffuser to dynamically explore and define the target class space. The methodology consists of two core phases:

1. Exploration of the Target Class Space

Rather than assuming a fixed number of classes, the model employs a Hierarchical Class Exploration strategy. Utilizing the Metropolis-Hastings (M-H) framework, the Unknown Diffuser statistically decides whether a cluster is too diverse and should be split into two, or if two clusters are sufficiently similar to be merged. This allows the model to discover a more accurate class space that reflects the true distribution of both known and unknown categories.

2. Knowledge Transfer and Generalization

Once the class space is estimated, the model applies Contrastive Learning to refine its representations. It identifies high-confidence "known" samples and pulls similar predictions closer while pushing dissimilar ones away. This process strengthens the model's memory of source knowledge while allowing it to generalize effectively to the newly discovered unknown clusters. The final optimization balances alignment loss with knowledge transfer loss derived from clustered pseudo-labels.

The UPUK framework illustrating the exploration of target class space and subsequent knowledge transfer and generalization.

Experimental Results

The UPUK model was evaluated on three standard domain adaptation benchmarks: Office-31, Office-Home, and VisDA-C 2017. The primary evaluation metric is the Harmonic Mean (HOS) of known and unknown class accuracy, ensuring the model does not achieve high scores by simply ignoring one group.

  • Office-Home: On this challenging 65-class benchmark, UPUK achieved an average HOS of 69.2%, a significant improvement over previous state-of-the-art methods such as AaD (63.6%) and SHOT (45.6%).
  • Office-31: The method reached a HOS of 79.7%, outperforming AaD's 77.4%.
  • Qualitative Analysis: t-SNE visualizations demonstrate that the proposed method creates much tighter and more distinct clusters for both known and unknown classes compared to traditional approaches, confirming the effectiveness of the dynamic class discovery.

Conclusion

This research demonstrates that "unknowns help knowns" in the context of OS-SFDA. By moving away from the traditional single-sink-class approach and employing a dynamic Unknown Diffuser, the UPUK framework achieves superior knowledge transfer and generalization. While the Metropolis-Hastings framework introduces some computational overhead, the gains in accuracy and cluster reliability represent a significant advancement for source-free adaptation in open environments. Future work will explore linking these discovered clusters to external knowledge bases for semantic identification of unknown classes.

Summary updated

Abstract

Open-Set Source-Free Domain Adaptation aims to transfer knowledge in realistic scenarios where the target domain has additional unknown classes compared to the limited-access source domain. Due to the absence of information on unknown classes, existing methods mainly transfer knowledge of known classes while roughly grouping unknown classes as one, attenuating the knowledge transfer and generalization. In contrast, this paper advocates that exploring unknown classes can better identify known ones, and proposes a domain adaptation model to transfer knowledge on known and unknown classes jointly. Specifically, given a source pre-trained model, we first introduce an unknown diffuser that can determine whether classes in space need to be split and merged through similarity measures, to estimate and generate a wider class space distribution, including known and unknown classes. Based on such a wider space distribution, we enhance the reliability of known class knowledge in the source pre-trained model through contrastive constraint. Finally, various supervision information, including reliable known class knowledge and clustered pseudo-labels, optimize the model for impressive knowledge transfer and generalization. Extensive experiments show that our network can achieve superior exploration and knowledge generalization on unknown classes, while with excellent known class transfer. The code is available at https://github.com/xdwfl/UPUK.
BibTeX
@inproceedings{wan2024unveiling,
  author = {Fuli Wan and Han Zhao and Xu Yang and Cheng Deng},
  title = {{Unveiling the Unknown: Unleashing the Power of Unknown to Known in Open-Set Source-Free Domain Adaptation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=QSDB9QXk10},
}
Figure