Cross-Inferential Networks for Source-Free Unsupervised Domain Adaptation

Yushun Tang, Qinghai Guo, Zhihai He

Abstract

Source-free Unsupervised Domain Adaptation (SF-UDA) presents a significant challenge in machine learning: adapting a pre-trained model to a new target domain without access to the original source data. This paper introduces Cross-Inferential Networks (CIN), a framework that utilizes a secondary "examiner" network to provide a self-reinforcing feedback loop. By shifting the task from simple classification to relative ordering, CIN effectively validates pseudo-labels and achieves state-of-the-art performance across multiple benchmarks.

1. Introduction

Deep learning models are notoriously sensitive to domain drift, where a model trained on a controlled source domain fails to generalize to a different target domain. While standard Unsupervised Domain Adaptation (UDA) addresses this by co-training on both datasets, real-world constraints—such as data privacy regulations (GDPR, HIPAA) or the logistical burden of transferring massive datasets—often make source data unavailable.

This necessitates Source-free UDA (SF-UDA). The primary hurdle in this setting is the lack of ground truth or source anchors to validate the model's predictions on the target data. Without a mechanism to verify "pseudo-labels," models often succumb to error accumulation. Cross-Inferential Networks (CIN) address this by introducing a dual-network architecture that cross-references classification logic with spatial and relational reasoning.

2. Methodology

2.1 Dual-Network Architecture

The CIN framework consists of two primary components:

  1. The Base Network ($\Phi$): A standard image classifier pre-trained on the source domain. Its goal is to assign categorical labels to target images.
  2. The Examiner Network ($\Gamma$): A specialized module designed for "relative ordering." Instead of classifying an image, the Examiner evaluates triplets (Anchor, Sample B, Sample C) to determine if Sample B is more similar to the Anchor than Sample C is.

By employing a different task (relational vs. categorical), the Examiner provides a "second opinion" that is less likely to share the same systematic biases as the Base Network.

2.2 Collaborative Adaptation and Consistency

The adaptation process is governed by two novel consistency constraints that force the networks to align their internal logic:

  • Correlation Matrix Consistency: This ensures that the feature similarity patterns identified by the Base Network’s feature extractor are mathematically consistent with the relative ordering results produced by the Examiner.
  • Attention Consistency: To ensure the networks are making decisions based on relevant features, both are forced to focus on the same salient regions. For example, if the Base Network identifies a "dog" by looking at the ears, the Examiner must also prioritize those regions when determining similarity.

2.3 Curriculum Sampling

To mitigate the risk of learning from incorrect pseudo-labels, CIN employs a curriculum-based approach. The Examiner initially trains only on high-confidence samples from the Base Network. As the system's overall accuracy improves, the threshold is lowered, allowing the model to learn from increasingly complex and ambiguous cases.

3. Experimental Results

The authors evaluated CIN on three major domain adaptation benchmarks: Office-31, Office-Home, and the large-scale VisDA-2017 dataset.

3.1 Performance Benchmarks

CIN demonstrated superior performance compared to existing state-of-the-art methods like SHOT:

  • VisDA-2017: CIN achieved an accuracy of 86.7%, representing a 3.8% improvement over the SHOT baseline. This is particularly notable given the synthetic-to-real nature of the dataset.
  • Office-Home: The model reached an average accuracy of 73.0%, setting a new benchmark for source-free adaptation on this dataset.

3.2 Ablation Study

Ablation experiments confirmed that both consistency mechanisms are essential. Removing Attention Consistency or Correlation Consistency resulted in a measurable decline in accuracy, suggesting that the "where" (attention) and the "what" (features) are both critical for robust adaptation.

4. Discussion and Limitations

While CIN offers a robust solution for SF-UDA, it introduces additional computational overhead due to the training of the secondary Examiner network. Furthermore, the success of the curriculum sampling is dependent on the initial source model possessing a baseline level of competency. If the initial domain gap is too wide to generate reliable high-confidence samples, the adaptation process may struggle to initiate.

5. Conclusion

Cross-Inferential Networks provide a novel pathway for source-free adaptation by leveraging task diversity. By forcing a classifier to justify its predictions through a secondary relational task, CIN creates a self-supervised environment that effectively bridges the gap between domains without the need for original source data. Future work may explore pre-training the Examiner on source data where available to further enhance the initial adaptation phase.

Summary updated

Abstract

One central challenge in source-free unsupervised domain adaptation (UDA) is the lack of an effective approach to evaluate the prediction results of the adapted network model in the target domain. To address this challenge, we propose to explore a new method called cross-inferential networks (CIN). Our main idea is that, when we adapt the network model to predict the sample labels from encoded features, we use these prediction results to construct new training samples with derived labels to learn a new examiner network that performs a different but compatible task in the target domain. Specifically, in this work, the base network model is performing image classification while the examiner network is tasked to perform relative ordering of triplets of samples whose training labels are carefully constructed from the prediction results of the base network model. Two similarity measures, cross-network correlation matrix similarity and attention consistency, are then developed to provide important guidance for the UDA process. Our experimental results on benchmark datasets demonstrate that our proposed CIN approach can significantly improve the performance of source-free UDA.
BibTeX
@inproceedings{tang2023cross,
  author = {Yushun Tang and Qinghai Guo and Zhihai He},
  title = {{Cross-Inferential Networks for Source-Free Unsupervised Domain Adaptation}},
  booktitle = {ICIP},
  year = {2023},
  url = {https://openreview.net/forum?id=uuHq5wG9EY},
}
Figure