Learning causal representations for robust domain adaptation

Shuai Yang, Kui Yu, Fuyuan Cao, Lin Liu, Hao Wang, Jiuyong LiIEEE TKDE 2020 · DOI · OpenReview

TL;DR

This paper introduces a novel "Causal AutoEncoder" (CAE) for robust domain adaptation, a scenario where target domain data is completely unavailable during training. CAE learns domain-invariant causal representations from a single source domain by combining deep autoencoders with causal structure learning, outperforming existing methods on several benchmarks.

What Problem Does This Paper Address?

Traditional machine learning assumes training and testing data come from the same distribution. When this assumption is violated (domain shift), domain adaptation techniques are used. Most existing methods require unlabeled target domain data to learn domain-invariant features. However, in many real-world scenarios, target domain data is scarce or entirely unavailable. This paper tackles the challenging problem of robust domain adaptation, where only labeled source domain data is available for training, and the goal is to generalize to an unseen target domain. The core challenge is to learn representations that are robust to domain shifts without any information from the target domain.

Methods

The authors propose a Causal AutoEncoder (CAE), a unified model that integrates two key components:

  1. Deep Autoencoder: This component learns low-dimensional representations of the input data. This dimensionality reduction helps to denoise the data and capture essential features.
  2. Causal Structure Learning: This component aims to disentangle the learned low-dimensional representations into two groups:
  3. Causal Representations (MB(Y)): These are identified as the Markov Blanket (MB) of the class variable. The MB is a minimal set of features that are causally related to the target variable and are assumed to be invariant across domains.
  4. Task-Irrelevant Representations (MB(Y)): These are features that are not part of the Markov Blanket and are considered less important or potentially domain-specific.

CAE optimizes an objective function that includes:

  • Reconstruction Loss ($L_D$): To ensure the autoencoder can reconstruct the input from the learned representations.
  • Causal Structure Loss ($L_C$): To enforce the separation of representations into causal and task-irrelevant groups, guided by causal discovery principles.
  • Cross-Entropy Loss ($L_y$): To leverage the label information from the source domain and train a classifier on the causal representations.
  • Regularization Loss ($L_R$): To prevent overfitting.

The causal structure learning is achieved by modeling the relationships between the learned representations and the class label as a Directed Acyclic Graph (DAG) and enforcing acyclicity constraints.

Datasets

The proposed CAE method was evaluated on three commonly used domain adaptation datasets:

  • Office-Caltech10: A visual domain adaptation dataset with 2,533 images across 10 categories from four domains (Caltech-256, Amazon, Webcam, DSLR). Experiments were conducted on 12 cross-domain tasks.
  • Amazon Review: A cross-domain sentiment analysis dataset comprising product reviews from Amazon. Experiments were conducted on 12 cross-domain tasks across four product categories (Books, DVDs, Electronics, Kitchen appliances).
  • Reuters-21578: A collection of news documents organized hierarchically. Experiments were conducted on 3 cross-domain tasks using the 500 most frequent terms as features.

Key Insights and Ideas

  • Causality for Robustness: The core idea is that causal relationships between features and the class variable are more likely to be invariant across different domains than mere correlations. This makes causal features ideal for robust domain adaptation.
  • Learning Without Target Data: The paper demonstrates that it's possible to learn effective domain-invariant representations solely from labeled source data, a significant departure from most existing domain adaptation methods.
  • Markov Blanket as Causal Features: Identifying the Markov Blanket of the class variable is proposed as a principled way to extract these domain-invariant causal features.
  • Unified Causal AutoEncoder: Combining deep autoencoders for representation learning with causal structure learning in a single framework is a novel approach that addresses the limitations of both individual techniques. Autoencoders reduce noise and dimensionality, while causal learning ensures the extracted features are robust.

Key Results

CAE consistently outperformed eleven state-of-the-art methods (including correlation-based feature selection, causal feature selection, domain adaptation, and stable learning methods) across all three datasets.

  • Office-Caltech10: CAE achieved the best average rank (1.46), significantly outperforming other methods.
  • Amazon Review: CAE achieved the best average rank (1.17), demonstrating strong performance.
  • Reuters-21578: CAE achieved the best average rank (2.00), again showing superiority.

The ablation studies also confirmed the importance of both the causal structure loss ($L_C$) and the cross-entropy loss ($L_y$) for CAE's performance.

Limitations and Future Work

  • Sensitivity to Hyperparameters: The paper acknowledges that CAE is sensitive to its hyperparameters (λ₁, λ₂, λ₃, l, and k), and careful tuning is required for optimal performance.
  • Future Work: The authors suggest combining autoencoders with other deep neural networks to learn even higher-quality causal representations for robust domain adaptation. This could involve exploring more complex network architectures or different causal discovery algorithms.

Summary updated

Abstract

Domain adaptation solves the learning problem in a target domain by leveraging the knowledge in a relevant source domain. While remarkable advances have been made, almost all existing domain adaptation methods heavily require large amounts of unlabeled target domain data for learning domain invariant representations to achieve good generalizability on the target domain. In fact, in many real-world applications, target domain data may not always be available. In this paper, we study the cases where at the training phase the target domain data is unavailable and only well-labeled source domain data is available, called robust domain adaptation. To tackle this problem, under the assumption that causal relationships between features and the class variable are robust across domains, we propose a novel Causal AutoEncoder (CAE), which integrates deep autoencoder and causal structure learning into a unified model to learn causal representations only using data from a single source domain. Specifically, a deep autoencoder model is adopted to learn low-dimensional representations, and a causal structure learning model is designed to separate the low-dimensional representations into two groups: causal representations and task-irrelevant representations. Using three real-world datasets the extensive experiments have validated the effectiveness of CAE compared to eleven state-of-the-art methods.
BibTeX
@article{yang2020learning,
  author = {Shuai Yang and Kui Yu and Fuyuan Cao and Lin Liu and Hao Wang and Jiuyong Li},
  title = {{Learning causal representations for robust domain adaptation}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2020},
  doi = {10.1109/tkde.2021.3119185},
}
Figure