Mutual Learning of Joint and Separate Domain Alignments for Multi-Source Domain Adaptation

Yuanyuan Xu, Meina Kan, Shiguang Shan, Xilin Chen

Abstract

Multi-Source Domain Adaptation (MSDA) aims to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Existing methods typically employ either joint alignment, which treats all sources as a single entity, or separate alignment, which aligns each source to the target individually. This paper introduces the Mutual Learning based Alignment Network (MLAN), a framework that integrates both strategies through a mutual learning module. By leveraging categorical and logits-based mutual learning, MLAN allows joint and separate branches to collaboratively refine predictions. Experimental results on benchmarks like DomainNet and Office-31 demonstrate that MLAN achieves state-of-the-art performance, particularly on complex datasets with significant domain gaps.

Introduction

In machine learning, the assumption that training and testing data share the same distribution often fails in practice. Unsupervised Domain Adaptation (UDA) addresses this by transferring knowledge from a labeled source domain to an unlabeled target domain. Multi-Source Domain Adaptation (MSDA) extends this to multiple source domains, introducing complexities such as domain gaps between sources and the target, as well as discrepancies among the sources themselves.

Current MSDA strategies generally fall into two categories:

  1. Joint Alignment: Aligns all source domains collectively with the target. While it captures shared information, it may lose domain-specific details.
  2. Separate Alignment: Aligns each source to the target individually. This preserves fine-grained information but neglects inter-source relationships.

The proposed MLAN framework seeks to harmonize these approaches, capturing both the "big picture" of joint alignment and the "fine details" of separate alignment.

Methodology: The MLAN Framework

The Mutual Learning based Alignment Network (MLAN) consists of three primary components designed to work in tandem.

Overview of the MLAN architecture, featuring joint and separate alignment branches coupled by a mutual learning module.

1. Joint Alignment Branch

This branch treats the collection of source domains as a single entity. It utilizes a shared feature extractor and classifier to identify features that are invariant across all domains, providing a consensus-based prediction.

2. Separate Alignment Branch

Comprising $N$ sub-branches (one for each source), this component aligns each individual source domain with the target. The final prediction is a weighted ensemble, where weights are determined by source-target similarity and prediction certainty.

3. Mutual Learning Module

The core innovation of MLAN is the mutual learning module, which facilitates communication between the joint and separate branches:

  • Categorical Mutual Learning: For high-confidence samples where both branches agree, the model employs source-guided K-means clustering to generate pseudo-labels for additional supervision.
  • Logits Mutual Learning: For low-confidence samples, the model uses mutual distillation. The more "certain" branch (lower entropy) serves as a teacher for the other on a per-sample basis.

Experimental Evaluation

MLAN was evaluated on three major MSDA benchmarks:

  • DomainNet: A large-scale dataset with ~0.6 million images across 6 domains and 345 categories.
  • Office-31: A standard dataset with 3 domains (Amazon, Webcam, DSLR) and 31 categories.
  • Digits-five: A collection of five digit datasets (MNIST, SVHN, USPS, etc.).

Results and Discussion

MLAN achieved state-of-the-art results across all tested benchmarks. On the challenging DomainNet dataset, MLAN reached an average accuracy of 53.5%, outperforming both joint-alignment (e.g., DRT+ST) and separate-alignment (e.g., DCTN) baselines.

Performance comparison on the DomainNet dataset. MLAN achieves the highest average accuracy across six target domains.

On Office-31, the model achieved 91.4% average accuracy, with significant gains in the "Transfer to Amazon" task. Despite the dual-branch architecture, MLAN remains computationally efficient, with only a marginal increase in parameters compared to single-strategy models. The results suggest that the mutual learning strategy effectively leverages the strengths of both alignment paradigms.

Conclusion

MLAN demonstrates that joint and separate alignment strategies are complementary rather than mutually exclusive. By integrating them through a dynamic mutual learning framework, MLAN captures a richer set of features for domain adaptation. Future work may explore handling more extreme domain shifts or scenarios with non-overlapping category sets.

Summary updated

Abstract

Multi-Source Domain Adaptation (MSDA) aims at transferring knowledge from multiple labeled source domains to benefit the task in an unlabeled target domain. The challenges of MSDA lie in mitigating domain gaps and combining information from diverse source domains. In most existing methods, the multiple source domains can be jointly or separately aligned to the target domain. In this work, we consider that these two types of methods, i.e. joint and separate domain alignments, are complementary and propose a mutual learning based alignment network (MLAN) to combine their advantages. Specifically, our proposed method is composed of three components, i.e. a joint alignment branch, a separate alignment branch, and a mutual learning objective between them. In the joint alignment branch, the samples from all source domains and the target domain are aligned together, with a single domain alignment goal, while in the separate alignment branch, each source domain is individually aligned to the target domain. Finally, by taking advantage of the complementarity of joint and separate domain alignment mechanisms, mutual learning is used to make the two branches learn collaboratively. Compared with other existing methods, our proposed MLAN integrates information of different domain alignment mechanisms and thus can mine rich knowledge from multiple domains for better performance. The experiments on Domain-Net, Office-31, and Digits-five datasets demonstrate the effectiveness of our method.
BibTeX
@inproceedings{xu2022mutual,
  author = {Yuanyuan Xu and Meina Kan and Shiguang Shan and Xilin Chen},
  title = {{Mutual Learning of Joint and Separate Domain Alignments for Multi-Source Domain Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00172},
}
Figure