Domain-Invariant Representation Learning with Global and Local Consistency

Wenwen Qiang, Jiangmeng Li, Jie Hu, Bing Su, Changwen Zheng, Hui XiongOpenReview 2021 · OpenReview

TL;DR

This paper tackles a key limitation in unsupervised domain adaptation: learned representations for target data often lack discriminability. They introduce a novel framework and a method called RLGLC, which uses a new Wasserstein distance variant and a regularized contrastive loss to improve target domain representation discriminability, achieving state-of-the-art results.

What Problem Does This Paper Address?

Unsupervised Domain Adaptation (UDA) aims to train models on labeled source data and apply them to unlabeled target data from a different distribution. A common approach is representation learning, where the goal is to learn features that are both discriminative (useful for the task) and transferable (work across domains). However, existing methods often achieve transferability and compressibility for target domain features but fail to adequately improve their discriminability. This means the model might struggle to distinguish between classes in the target domain, even if it can generalize. This paper addresses this gap by proposing a framework and method to explicitly enhance the discriminability of target domain representations.

Methods

The core of the paper is the Domain-Invariant Representation Learning with Global and Local Consistency (RLGLC) method. It's built upon an information-theoretic analysis of existing UDA frameworks, revealing that they don't fully guarantee target domain discriminability.

The proposed framework introduces two key components:

  • Global Consistency Module (GCM): This module aims to align the distributions of the source domain and the target domain, as well as the target domain and a "self-supervised" augmented version of itself. It introduces a novel metric called Asymmetrically-Relaxed Wasserstein of Wasserstein Distance (AR-WWD). AR-WWD uses the Wasserstein distance as a ground metric, which is known to be robust to image transformations, and imposes an inequality constraint to ensure the source domain distribution is contained within the target domain's distribution. This helps maintain transferability and compressibility.
  • Local Consistency Module (LCM): This module focuses on improving target domain discriminability. It uses a Regularized Contrastive Loss (RCL). While standard contrastive losses push dissimilar samples apart, RCL adds a regularization term to prevent semantically similar instances from being undesirably separated, thus preserving important predictive information.

The overall objective function combines distribution alignment (using AR-WWD), classification loss on the source domain, and the regularized contrastive loss for the target domain.

Datasets

The paper evaluates RLGLC on several standard UDA benchmarks:

  • Office-Home: Contains 4 domains.
  • Office-31: Contains 3 domains.
  • VisDA-2017: Contains 12 categories.
  • Digits datasets: Includes SVHN, USPS, and MNIST.

Key Insights and Ideas

  • Target Discriminability is Crucial: The paper highlights that existing UDA methods, while good at aligning distributions, often neglect the discriminability of target domain representations, which is essential for accurate classification.
  • Information Theory as a Guide: The authors leverage information theory to analyze the limitations of current UDA frameworks and motivate their new approach.
  • AR-WWD for Robust Alignment: The proposed AR-WWD metric offers a more robust way to align distributions, leveraging the perceptual qualities of Wasserstein distance and an inequality constraint for better transferability.
  • RCL for Preserving Semantic Similarity: The regularized contrastive loss is a clever way to achieve discriminative representations without sacrificing the relationships between semantically similar target samples.
  • Theoretical Guarantees: The paper provides theoretical analysis linking their framework to reduced Bayes error rates, strengthening the claims of effectiveness.

Key Results

RLGLC achieves state-of-the-art performance across multiple UDA benchmarks.

  • On Office-31, RLGLC achieved the highest average classification accuracy and won the most specific transfer tasks.
  • On Office-Home, RLGLC also achieved the highest average result and won the majority of specific transfer tasks.
  • On VisDA-2017, RLGLC showed significant improvement, outperforming previous methods.
  • On Digits datasets, RLGLC exceeded all compared approaches in average accuracy and demonstrated near state-of-the-art performance on most specific transfer tasks.

The ablation studies also showed that both the global consistency module (AR-WWD) and the local consistency module (RCL) contribute significantly to the overall performance.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper includes a hyperparameter sensitivity study (Figure 1), indicating that the performance can be influenced by the choices of $\alpha$ and $\beta$. While they find optimal values, further research into more robust hyperparameter selection could be beneficial.
  • Computational Cost: While not explicitly detailed as a limitation, the use of Wasserstein distance and contrastive learning can sometimes be computationally intensive. Future work might explore more efficient implementations or approximations.
  • Further Theoretical Exploration: The paper provides theoretical analysis related to Bayes error rate. Expanding this analysis to cover other aspects of the learned representations or exploring different theoretical frameworks could be valuable.

Summary updated

Abstract

In this paper, we give an analysis of the existing representation learning framework of unsupervised domain adaptation and show that the learned feature representations of the source domain samples are with discriminability, compressibility, and transferability. However, the learned feature representations of the target domain samples are only with compressibility and transferability. To address this challenge, we propose a novel framework and show from the information theory view that this framework can effectively improve the discriminability of the target domain sample representation. We also propose a method, namely domain-invariant representation learning with global and local consistency (RLGLC), under this framework. In particular, to maintain the global consistency, RLGLC proposes a new metric called asymmetrically-relaxed Wasserstein of Wasserstein distance (AR-WWD), AR-WWD can not only extract the transferability and compressibility of the feature representation of two domains, but also correlates well with human perception. To impose the local consistency structures, we propose a regularized contrastive loss, which can not only keep as much as possible predictive information contained in the feature representation of the target domain, but also alleviates the problem that semantically similar instances are undesirable pushed apart in training processing. Finally, we verify the effectiveness of RLGLC from both theoretical analyses on Bayes error rate and experimental validation on several benchmarks.
BibTeX
@misc{qiang2021domain,
  author = {Wenwen Qiang and Jiangmeng Li and Jie Hu and Bing Su and Changwen Zheng and Hui Xiong},
  title = {{Domain-Invariant Representation Learning with Global and Local Consistency}},
  year = {2021},
  url = {https://openreview.net/forum?id=pXNXwaLu5MN},
  howpublished = {OpenReview},
}
Figure