Domain Adaptation with Cauchy-Schwarz Divergence

Wenzhe Yin, Shujian Yu, Yicong Lin, Jie Li, Jan-Jakob Sonke, Efstratios Gavves

Abstract

Domain adaptation aims to use training data from one or multiple source domains to learn a hypothesis that can be generalized to a different, but related, target domain. As such, having a reliable measure for evaluating the discrepancy of both marginal and conditional distributions is crucial. We introduce Cauchy-Schwarz (CS) divergence to the problem of unsupervised domain adaptation (UDA). The CS divergence offers a theoretically tighter generalization error bound than the popular Kullback-Leibler divergence. This holds for the general case of supervised learning, including multi-class classification and regression. Furthermore, we illustrate that the CS divergence enables a simple estimator on the discrepancy of both marginal and conditional distributions between source and target domains in the representation space, without requiring any distributional assumptions. We provide multiple examples to illustrate how the CS divergence can be conveniently used in both distance metric- or adversarial training-based UDA frameworks, resulting in compelling performance.

Introduction

In standard machine learning, we typically assume that training (source) and testing (target) data are drawn from the same distribution. However, real-world applications often face domain shift—discrepancies caused by variations in lighting, camera angles, or backgrounds. Unsupervised Domain Adaptation (UDA) seeks to mitigate this by learning domain-invariant features.

While many methods focus on matching marginal distributions, they often neglect the conditional shift, where the relationship between features and labels changes across domains. Traditional metrics like Kullback-Leibler (KL) divergence can suffer from numerical instability when probability densities approach zero, while Maximum Mean Discrepancy (MMD) may lack the statistical power to distinguish complex conditional distributions. This paper proposes the Cauchy-Schwarz (CS) divergence as a robust alternative.

Methodology

The proposed approach leverages the Cauchy-Schwarz inequality to define a divergence measure that is both theoretically grounded and computationally efficient.

Theoretical Advantages

The authors prove that CS divergence provides a tighter generalization error bound than KL divergence. Minimizing this bound ensures that a model trained on the source domain is more likely to generalize effectively to the target domain. Unlike KL divergence, which involves a ratio ($p/q$) that can explode, CS divergence utilizes an inner-product-like structure, ensuring numerical stability.

Framework and Estimators

The method employs non-parametric, kernel-based estimators for both marginal CS and Conditional CS (CCS). These estimators do not require expensive matrix inversions or specific distributional assumptions (e.g., Gaussianity). The authors integrate these measures into two primary training frameworks: a distance-metric-based approach (CS+CCS) and an adversarial framework (CS-adv).

The proposed adversarial training framework utilizing CS and CCS divergences for domain alignment.

In the adversarial setup, a feature extractor $f$ learns to map inputs to a latent space where a discriminator (comprising two classifiers $g_1, g_2$) cannot distinguish between source and target samples. The CS and CCS losses directly guide this alignment in the representation space.

Experimental Results

The effectiveness of CS divergence was evaluated across several standard UDA benchmarks, including Digits, Office-31, Office-Home, and VisDA17.

Performance Benchmarks

  • Office-Home: The CS-adv method achieved an average accuracy of 71.2%, significantly outperforming baselines like CDAN (65.8%).
  • Office-31: Reached a state-of-the-art average of 90.0%.
  • VisDA17: Demonstrated a substantial jump in performance, moving from 52.3% (KL-based) to 64.5% (CS+CCS).

Ablation Study

A key insight from the ablation studies is the necessity of matching both marginal and conditional distributions. As shown in the training curves, combining CS and CCS consistently yields the highest accuracy.

Ablation study showing the performance gains of combining CS and CCS compared to standard MMD and individual components.

Conclusion

Cauchy-Schwarz divergence offers a powerful, stable, and theoretically superior alternative to KL divergence for UDA. By effectively aligning both marginal and conditional distributions without restrictive assumptions, the proposed framework achieves competitive results on diverse computer vision tasks. Future work includes extending this divergence to regression and object detection, as well as exploring its utility in source-private domain adaptation scenarios.

Summary updated

Abstract

Domain adaptation aims to use training data from one or multiple source domains to learn a hypothesis that can be generalized to a different, but related, target domain. As such, having a reliable measure for evaluating the discrepancy of both marginal and conditional distributions is crucial. We introduce Cauchy-Schwarz (CS) divergence to the problem of unsupervised domain adaptation (UDA). The CS divergence offers a theoretically tighter generalization error bound than the popular Kullback-Leibler divergence. This holds for the general case of supervised learning, including multi-class classification and regression. Furthermore, we illustrate that the CS divergence enables a simple estimator on the discrepancy of both marginal and conditional distributions between source and target domains in the representation space, without requiring any distributional assumptions. We provide multiple examples to illustrate how the CS divergence can be conveniently used in both distance metric- or adversarial training-based UDA frameworks, resulting in compelling performance.
BibTeX
@inproceedings{yin2024domain,
  author = {Wenzhe Yin and Shujian Yu and Yicong Lin and Jie Li and Jan-Jakob Sonke and Efstratios Gavves},
  title = {{Domain Adaptation with Cauchy-Schwarz Divergence}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=62m7yvkGIY},
}
Figure