Deep Least Squares Alignment for Unsupervised Domain Adaptation

Youshan Zhang, Brian D. Davison

TL;DR

This paper introduces Deep Least Squares Alignment (DLSA), a novel approach to Unsupervised Domain Adaptation (UDA) that explicitly models domain distributions in a latent space using linear fitting lines. By aligning these lines through marginal and conditional adaptation losses, DLSA effectively reduces domain discrepancy and achieves state-of-the-art performance on several benchmark datasets.

What Problem Does This Paper Address?

Unsupervised Domain Adaptation (UDA) aims to train accurate models on a target domain with limited or no labeled data, by leveraging a richly labeled source domain. A key challenge in UDA is the "domain shift" or "dataset bias," where the source and target domains have different characteristics, leading to performance degradation. Existing deep learning methods for UDA often focus on aligning the overall distributions of the two domains, but they can struggle to intuitively represent these distributions and their alignment process. Furthermore, they may not fully consider the latent structure of the target domain or effectively align both marginal and conditional distributions. This paper addresses the need for a more intuitive and effective way to model and align domain distributions in UDA.

Methods

The core of DLSA lies in its novel approach to modeling domain distributions. Instead of just aligning overall distributions, DLSA explicitly models the distribution of each domain in a latent space using least squares estimation to find a linear fitting line. This line is parameterized by its slope and intercept.

The method has two main contributions:

  1. Explicit Distribution Modeling: DLSA estimates the latent space distribution of both source and target domains by fitting a linear model (y = ax + b) to the extracted features. The slope and intercept of these lines are used as key parameters to represent the domain's distribution.
  2. Marginal and Conditional Adaptation Losses:
    • Marginal Adaptation Loss ($L_M$): This loss aims to align the overall distributions of the source and target domains. It minimizes the difference between the slopes (angle) and intercepts of their respective fitting lines. This encourages domain invariance by making the global angle and translation of the fitting lines similar.
  3. Conditional Adaptation Loss ($L_C$): This loss refines the alignment by considering class-specific distributions. It uses pseudo-labels generated for the target domain and minimizes the angle and intercept differences between fitting lines for each class. This ensures that not only the overall domain distributions but also the class-conditional distributions are aligned.

The overall objective function combines the standard source classification loss ($L_S$) with these marginal and conditional adaptation losses, balanced by hyperparameters.

Datasets

The proposed DLSA model was evaluated on the following standard UDA benchmark datasets:

  • Office + Caltech-10: Contains 2,533 images across four domains (Amazon, Webcam, DSLR, Caltech) and ten classes. Twelve tasks were evaluated.
  • Office-31: Contains 4,110 images across three domains (Amazon, Webcam, DSLR) and 31 classes. Six tasks were evaluated.
  • Office-Home: Contains 15,588 images across four domains (Art, Clipart, Product, Real-World) and 65 classes. Twelve tasks were evaluated.
  • VisDA-2017: A challenging dataset with a large domain shift between synthetic images and real-world images, featuring 12 classes. The evaluation was performed on a synthetic-to-real setting.

Key Insights and Ideas

  • Linearity as a Distribution Proxy: The paper posits that a simple linear relationship (slope and intercept) in the latent space can effectively capture and represent the distribution of a domain. This is a novel way to model domain characteristics for adaptation.
  • Geometric Alignment: By minimizing the angle and intercept differences between fitting lines, DLSA performs a geometric alignment of domain distributions. This provides a more intuitive understanding of the adaptation process compared to abstract distribution matching.
  • Marginal and Conditional Alignment Synergy: The combination of marginal and conditional adaptation losses is crucial. Marginal alignment provides a global domain invariance, while conditional alignment refines this by ensuring class-specific alignment, leading to better overall performance.
  • Pseudo-labels as a Bridge: The use of soft pseudo-labels for the target domain is essential for enabling conditional adaptation, bridging the gap of missing labels.
  • DLSA as a Plug-in: The paper demonstrates that DLSA can be effectively combined with existing state-of-the-art UDA models (e.g., SHOT, CAN) to further boost their performance, highlighting its versatility.

Key Results

  • DLSA consistently achieves higher average accuracy than state-of-the-art methods across the evaluated datasets (Office + Caltech-10, Office-31, Office-Home).
  • On the challenging Office-31 dataset, DLSA achieved an average accuracy of 91.6%, outperforming other methods.
  • For difficult transfer tasks like W→A and D→A on Office-31, DLSA showed substantially better transferring ability.
  • In the VisDA-2017 dataset, DLSA also outperformed SOTA models.
  • Ablation studies showed that the conditional adaptation loss ($L_C$) is more important than the marginal adaptation loss ($L_M$) for improving performance, though both contribute significantly.
  • Visualizations (t-SNE) show that DLSA leads to more discriminative feature representations with increased inter-class dispersion and intra-class compactness after adaptation.

Limitations and Future Work

  • Linearity Assumption: The paper acknowledges that assuming a linear relationship in the latent space might be a limitation. Exploring non-linear relationships (e.g., polynomial) could potentially improve performance.
  • Pseudo-label Quality: The performance of DLSA can be affected by the quality of generated pseudo-labels, especially if there are many outlier labels in the target domain.
  • Future Directions: The authors suggest improving pseudo-label generation methods and exploring techniques for removing outlier labels in the target domain to further enhance performance.

Summary updated

Abstract

Unsupervised domain adaptation leverages rich information from a labeled source domain to model an unlabeled target domain. Existing methods attempt to align the cross-domain distributions. However, the statistical representations of the alignment of the two domains are not well addressed. In this paper, we propose deep least squares alignment (DLSA) to estimate the distribution of the two domains in a latent space by parameterizing a linear model. We further develop marginal and conditional adaptation loss to reduce the domain discrepancy by minimizing the angle between fitting lines and intercept differences and further learning domain invariant features. Extensive experiments demonstrate that the proposed DLSA model is effective in aligning domain distributions and outperforms state-of-the-art methods.
BibTeX
@inproceedings{zhang2021deep,
  author = {Youshan Zhang and Brian D. Davison},
  title = {{Deep Least Squares Alignment for Unsupervised Domain Adaptation}},
  booktitle = {BMVC},
  year = {2021},
}
Figure