Bures Joint Distribution Alignment with Dynamic Margin for Unsupervised Domain Adaptation

Yong Hui Liu, Chuan-Xian Ren, Xiao-Lin Xu, Ke-Kun HuangarXiv 2022 · DOI · arXiv

Abstract

Unsupervised Domain Adaptation (UDA) is a critical task in machine learning, aimed at transferring knowledge from a labeled source domain to an unlabeled target domain under distribution shifts. Traditional methods often focus on marginal or conditional distribution alignment in Euclidean space, which can fail to capture complex nonlinear structures and joint feature-label dependencies. This paper introduces Bures Joint Distribution Alignment (BJDA), a novel framework that aligns joint distributions within a Reproducing Kernel Hilbert Space (RKHS) using the Bures-Wasserstein distance. To further enhance discriminative power, we propose a Dynamic Margin Contrastive (DMC) loss that adaptively adjusts class margins based on model confidence. Experimental results across six benchmarks demonstrate that BJDA consistently outperforms existing state-of-the-art methods.

Introduction

The performance of deep learning models often degrades significantly when applied to data from a target domain that differs from the training source domain. This phenomenon, known as distribution shift, is a major hurdle for real-world deployment. Unsupervised Domain Adaptation (UDA) addresses this by leveraging labeled source data and unlabeled target data to learn domain-invariant representations.

While many UDA techniques align marginal distributions $P(X)$ or conditional distributions $P(Y|X)$, they frequently overlook the joint distribution $P(X, Y)$, which captures the essential relationship between features and labels. Moreover, performing alignment in simple Euclidean space often fails to resolve the intricate, nonlinear manifolds found in high-dimensional image data.

Methodology

Bures Joint Distribution Alignment (BJDA)

The core of the BJDA approach is the direct alignment of joint distributions in an infinite-dimensional kernel space. By mapping data into a Reproducing Kernel Hilbert Space (RKHS), the model can effectively "unfold" complex nonlinear structures that appear overlapping in lower-dimensional spaces.

The researchers employ Optimal Transport (OT) theory, specifically the kernel Bures-Wasserstein distance, to quantify the discrepancy between source and target joint distributions. Unlike previous methods that require iterative optimization, BJDA utilizes a closed-form solution for this distance, enabling efficient end-to-end training within a deep learning framework.

Comparison of distribution alignment strategies, highlighting the advantages of Bures Joint Distribution Alignment with Dynamic Margin.

Dynamic Margin Contrastive (DMC) Loss

To improve the separability of learned features, BJDA incorporates a novel contrastive learning component. Standard contrastive losses, such as the triplet loss, typically rely on a fixed margin parameter that requires extensive manual tuning.

The proposed DMC loss introduces a dynamic margin that scales according to the model's prediction confidence (measured via entropy). When the model is uncertain about a sample, a larger margin is enforced to push it further from incorrect category prototypes. This adaptive mechanism focuses the learning process on ambiguous samples and eliminates the need for a fixed, pre-defined margin.

Illustration of the proposed Dynamic Margin Contrastive (DMC) loss compared to the standard triplet loss.

Experimental Evaluation

The effectiveness of BJDA was validated on six diverse benchmarks, ranging from small-scale object datasets to massive real-world image collections including Adaptiope, Office-Home, DomainNet, and ImageCLEF-DA.

Key Results

BJDA achieved state-of-the-art (SOTA) results across the board. Notably:

  • On the Adaptiope dataset, BJDA improved average accuracy by 2.8% over previous leading methods.
  • On Office-Caltech10, it achieved a 1.4% improvement, reaching an average accuracy of 92.6%.
  • On ImageCLEF-DA, the model reached an accuracy of 91.2%, a 1.1% increase over the previous SOTA.

Ablation studies confirmed that both the Bures alignment and the dynamic margin are critical components. Removing either led to a measurable decline in performance, underscoring the synergy between joint distribution alignment in RKHS and confidence-driven contrastive learning.

Conclusion and Future Work

Bures Joint Distribution Alignment (BJDA) offers a robust solution for UDA by combining the geometric advantages of RKHS with adaptive contrastive learning. By aligning joint distributions and utilizing a dynamic margin, the model effectively handles complex distribution shifts without sensitive hyperparameter tuning. Future research will explore extending this framework to multi-source domain adaptation scenarios and further automating the balancing of different loss components.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) is one of the prominent tasks of transfer learning, and it provides an effective approach to mitigate the distribution shift between the labeled source domain and the unlabeled target domain. Prior works mainly focus on aligning the marginal distributions or the estimated class-conditional distributions. However, the joint dependency among the feature and the label is crucial for the adaptation task and is not fully exploited. To address this problem, we propose the Bures Joint Distribution Alignment (BJDA) algorithm which directly models the joint distribution shift based on the optimal transport theory in the infinite-dimensional kernel spaces. Specifically, we propose a novel alignment loss term that minimizes the kernel Bures-Wasserstein distance between the joint distributions. Technically, BJDA can effectively capture the nonlinear structures underlying the data. In addition, we introduce a dynamic margin in contrastive learning phase to flexibly characterize the class separability and improve the discriminative ability of representations. It also avoids the cross-validation procedure to determine the margin parameter in traditional triplet loss based methods. Extensive experiments show that BJDA is very effective for the UDA tasks, as it outperforms state-of-the-art algorithms in most experimental settings. In particular, BJDA improves the average accuracy of UDA tasks by 2.8% on Adaptiope, 1.4% on Office-Caltech10, and 1.1% on ImageCLEF-DA.
BibTeX
@misc{liu2022bures,
  author = {Yong Hui Liu and Chuan-Xian Ren and Xiao-Lin Xu and Ke-Kun Huang},
  title = {{Bures Joint Distribution Alignment with Dynamic Margin for Unsupervised Domain Adaptation}},
  year = {2022},
  eprint = {2203.06836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.06836},
}
Figure