Abstract
Domain adaptation aims to use training data from one or multiple source domains to learn a hypothesis that can be generalized to a different, but related, target domain. As such, having a reliable measure for evaluating the discrepancy of both marginal and conditional distributions is crucial. We introduce Cauchy-Schwarz (CS) divergence to the problem of unsupervised domain adaptation (UDA). The CS divergence offers a theoretically tighter generalization error bound than the popular Kullback-Leibler divergence. This holds for the general case of supervised learning, including multi-class classification and regression. Furthermore, we illustrate that the CS divergence enables a simple estimator on the discrepancy of both marginal and conditional distributions between source and target domains in the representation space, without requiring any distributional assumptions. We provide multiple examples to illustrate how the CS divergence can be conveniently used in both distance metric- or adversarial training-based UDA frameworks, resulting in compelling performance.
Introduction
In standard machine learning, we typically assume that training (source) and testing (target) data are drawn from the same distribution. However, real-world applications often face domain shift—discrepancies caused by variations in lighting, camera angles, or backgrounds. Unsupervised Domain Adaptation (UDA) seeks to mitigate this by learning domain-invariant features.
While many methods focus on matching marginal distributions, they often neglect the conditional shift, where the relationship between features and labels changes across domains. Traditional metrics like Kullback-Leibler (KL) divergence can suffer from numerical instability when probability densities approach zero, while Maximum Mean Discrepancy (MMD) may lack the statistical power to distinguish complex conditional distributions. This paper proposes the Cauchy-Schwarz (CS) divergence as a robust alternative.
Methodology
The proposed approach leverages the Cauchy-Schwarz inequality to define a divergence measure that is both theoretically grounded and computationally efficient.
Theoretical Advantages
The authors prove that CS divergence provides a tighter generalization error bound than KL divergence. Minimizing this bound ensures that a model trained on the source domain is more likely to generalize effectively to the target domain. Unlike KL divergence, which involves a ratio ($p/q$) that can explode, CS divergence utilizes an inner-product-like structure, ensuring numerical stability.
Framework and Estimators
The method employs non-parametric, kernel-based estimators for both marginal CS and Conditional CS (CCS). These estimators do not require expensive matrix inversions or specific distributional assumptions (e.g., Gaussianity). The authors integrate these measures into two primary training frameworks: a distance-metric-based approach (CS+CCS) and an adversarial framework (CS-adv).
In the adversarial setup, a feature extractor $f$ learns to map inputs to a latent space where a discriminator (comprising two classifiers $g_1, g_2$) cannot distinguish between source and target samples. The CS and CCS losses directly guide this alignment in the representation space.
Experimental Results
The effectiveness of CS divergence was evaluated across several standard UDA benchmarks, including Digits, Office-31, Office-Home, and VisDA17.
Performance Benchmarks
- Office-Home: The CS-adv method achieved an average accuracy of 71.2%, significantly outperforming baselines like CDAN (65.8%).
- Office-31: Reached a state-of-the-art average of 90.0%.
- VisDA17: Demonstrated a substantial jump in performance, moving from 52.3% (KL-based) to 64.5% (CS+CCS).
Ablation Study
A key insight from the ablation studies is the necessity of matching both marginal and conditional distributions. As shown in the training curves, combining CS and CCS consistently yields the highest accuracy.
Conclusion
Cauchy-Schwarz divergence offers a powerful, stable, and theoretically superior alternative to KL divergence for UDA. By effectively aligning both marginal and conditional distributions without restrictive assumptions, the proposed framework achieves competitive results on diverse computer vision tasks. Future work includes extending this divergence to regression and object detection, as well as exploring its utility in source-private domain adaptation scenarios.
Summary updated