A Unified Domain Adaptation Framework with Distinctive Divergence Analysis

Zhiri Yuan, Xixu Hu, Qi Wu, Shumin Ma, Cheuk Hang Leung, Xin Shen, Yiyan HuangTMLR 2022 · OpenReview

TL;DR

This paper introduces a unified theoretical framework for unsupervised domain adaptation (DA) that bridges the gap between existing DA models and their theoretical underpinnings. By proposing a generalized bound that accommodates various tasks (classification/regression) and divergence measures, the authors provide a solid theoretical foundation for a wide range of DA algorithms and introduce new model variants that demonstrate improved performance.

What Problem Does This Paper Address?

Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain, which often has a different data distribution. While theoretical bounds exist to explain why this transfer works, they are often too restrictive, typically limited to binary classification tasks and specific divergence measures like H-divergence. This has led to a proliferation of DA models that lack strong theoretical justification, making it difficult to understand why certain divergence measures work better than others or to systematically design new, effective DA algorithms. This paper addresses this gap by developing a unified theoretical framework that can encompass a broader range of DA tasks and divergence measures, thereby providing a more robust theoretical foundation and guiding future research.

Methods

The core of this paper is the development of a unified generalization bound for unsupervised domain adaptation. Unlike previous bounds that were specific to binary classification and H-divergence, this new bound is more general:

  • Task Agnostic: It applies to both classification and regression tasks.
  • Divergence Agnostic: It accommodates a wide range of distribution divergence measures, including f-divergences (like KL, JS, TV), Wasserstein distance, and MMD.

The authors achieve this by relaxing the constraints on the hypothesis set and focusing on the true source error rather than an approximation. They then demonstrate how this unified bound can theoretically recover many existing DA models and also inspire the design of new ones. A significant part of their work involves a comprehensive comparison of various divergence measures, analyzing their properties in terms of topology, convergence rate, and computational cost.

Datasets

The paper evaluates its proposed models and theoretical framework on several standard image datasets for domain adaptation tasks:

  • Office-Home: A dataset with 15,500 images across 65 object classes, forming four distinct domains (Artistic, Clipart, Product, Real-World).
  • Digits: Two datasets, MNIST and USPS, used for transfer tasks like M→U and U→M.
  • Office-31: A widely used dataset with 4,652 images across 31 categories, collected from three sources (Amazon, Webcam, DSLR), used for six transfer tasks.
  • MPI3D: A simulation-to-real dataset for image regression, consisting of three domains (Toy, Realistic, Real) with continuous factors to predict.
  • dSprites: A 2D synthetic dataset for image regression with three domains (Color, Noisy, Scream) and continuous factors to predict.

Key Insights and Ideas

  • Bridging the Theory-Practice Gap: The most significant insight is the creation of a unified theoretical framework that finally connects the diverse landscape of DA models with solid theoretical guarantees, moving beyond the limitations of earlier bounds.
  • Divergence Measure Matters: The paper highlights that while many divergence measures can be theoretically accommodated, their practical performance can vary significantly. The choice of divergence measure is not arbitrary and can have a substantial impact on model performance.
  • Topology vs. Performance: The analysis of divergence measures reveals a trade-off. Measures with finer topologies (like f-divergences) are theoretically appealing but can be harder to implement and estimate. Measures with weaker topologies (like MMD and Wasserstein) are often easier to implement and can achieve strong empirical results.
  • New Models from Theory: The unified framework not only explains existing models but also serves as a blueprint for designing new DA models, demonstrating that theoretical insights can directly lead to practical improvements.
  • Comprehensive Divergence Comparison: The paper provides a valuable resource by systematically comparing popular divergence measures across multiple dimensions, offering guidance for practitioners.

Key Results

The paper presents several key findings:

  • Improved Performance: The proposed new DA models, leveraging various divergence measures within the unified framework, achieve competitive or superior performance compared to state-of-the-art baselines on image classification and regression tasks. For instance, on the Office-Home dataset, a model using Pearson x² divergence achieved the highest average accuracy. On the MPI3D regression dataset, Total Variation divergence performed best among the proposed models.
  • Enriched Model Choices: The unified framework significantly expands the repertoire of theoretically grounded DA models, offering practitioners a wider selection of divergence measures to experiment with.
  • Theoretical Recovery: The framework successfully recovers and explains the theoretical underpinnings of several classical DA models, including MCD, AFN, and DAN.
  • Empirical Validation: Extensive experiments on benchmark datasets (Office-Home, Digits, Office-31, MPI3D, dSprites) validate the theoretical findings and demonstrate the effectiveness of the proposed models.

Limitations and Future Work

  • No Universal Best Divergence: The authors acknowledge that there isn't a single divergence measure that universally outperforms others across all tasks and datasets. This suggests that careful selection based on the specific problem is still necessary.
  • Open Question on Optimal Selection: While the paper provides a framework and comparisons, the question of how to definitively choose the "best" divergence measure for a given task remains an open area for future research.
  • Computational Cost: Some divergence measures, like MMD, can have high computational costs that grow quadratically with sample size, limiting scalability. The paper notes this as a factor influencing practical implementation.
  • Future Studies: The authors suggest that their work provides theoretical support for many existing models and sheds light on future studies of DA, implying that further exploration of novel divergence measures and their theoretical guarantees is warranted.

Summary updated

Abstract

Unsupervised domain adaptation enables knowledge transfer from a labeled source domain to an unlabeled target domain by aligning the learnt features of both domains. The idea is theoretically supported by the generalization bound analysis in Ben-David et al. (2007), which specifies the applicable task (binary classification) and designates a specific distribution divergence measure. Although most distribution-aligning domain adaptation models seek theoretical grounds from this particular bound analysis, they do not actually fit into the stringent conditions. In this paper, we bridge the long-standing theoretical gap in literature by providing a unified generalization bound. Our analysis can well accommodate the classification/regression tasks and most commonly-used divergence measures, and more importantly, it can theoretically recover a large amount of previous models. In addition, we identify the key difference in the distribution divergence measures underlying the diverse models and commit a comprehensive in-depth comparison of the commonly-used divergence measures. Based on the unified generalization bound, we propose new domain adaptation models that achieve transferability through domain-invariant representations and conduct experiments on real-world datasets that corroborate our theoretical findings. We believe these insights are helpful in guiding the future design of distribution-aligning domain adaptation algorithms.
BibTeX
@article{yuan2022unified,
  author = {Zhiri YUAN and Xixu HU and Qi WU and Shumin MA and Cheuk Hang LEUNG and Xin Shen and Yiyan HUANG},
  title = {{A Unified Domain Adaptation Framework with Distinctive Divergence Analysis}},
  journal = {Accepted by TMLR},
  year = {2022},
  url = {https://openreview.net/forum?id=yeT9cBq8Cn},
}

Loading related content...

Figure