Transportable Representations for Domain Generalization

Kasra Jalaldoust, Elias BareinboimAAAI 2024 · OpenReview

Abstract

This paper bridges the gap between causal transportability theory and representation learning to address the Domain Generalization (DG) problem. By introducing a formal framework to determine when a learned representation can reliably predict labels in unseen target domains, the authors prove that statistical invariance across source domains serves as a sound and complete criterion for generalization under specific causal stability assumptions.

Introduction

Most machine learning models rely on the "IID assumption"—the premise that training and testing data are drawn from the same distribution. In real-world applications, such as medical diagnostics across different hospitals, this assumption often fails, leading to Domain Generalization (DG) failures.

Existing DG methods generally fall into two categories: invariance-based approaches, which seek features that remain consistent across training sources, and causal-based approaches, which focus on the causal parents of a label. However, a unifying theory explaining why and when these methods succeed has been missing. This research addresses this gap by formally defining "transportable" representations that can be safely moved from one environment to another.

Methodology

The authors utilize Causal Transportability Theory to analyze representation learning through two primary lenses:

1. The Graphical Approach

The authors introduce Augmented Selection Diagrams. While standard selection diagrams use selection nodes ($S$) to indicate mechanism changes between domains, this framework augments these graphs by adding the representation ($R$) as a node linked to the variables it aggregates.

Augmented Selection Diagram showing the representation node R and selection nodes S.

Theorem 1 (Graphical Transportability): The authors provide a sound and complete graphical criterion to determine if a representation's predictive power is transportable to a target domain based on d-separation rules within this augmented graph.

2. The Data-Driven Approach

For scenarios where a perfect causal graph is unavailable, the paper proposes a graph-free alternative based on two key assumptions:

  • Stability of Mechanisms (SoM): If a mechanism generating a variable remains stable across all observed source domains, it is assumed to remain stable in the unseen target domain.
  • r-faithfulness: A technical assumption ensuring that observed statistical invariances reflect underlying causal stabilities rather than coincidental alignments.

Experimental Validation

The paper is primarily theoretical, using synthetically generated Structural Causal Models (SCMs) to validate its findings. These simulations allow for precise control over domain discrepancies, enabling a direct comparison between ground-truth target distributions and model predictions. The focus was on verifying mathematical soundness rather than optimizing for standard image classification benchmarks like PACS.

Key Insights

  • Causal isn't always "Best": A significant finding is that using only causal features (parents of the label $Y$) is not always optimal. A transportable representation can include non-causal features that provide higher predictive power, provided their relationship remains stable across domains.
  • The Duality of Invariance: The paper establishes a "graphical-invariance duality," proving that seeking invariant representations is mathematically equivalent to finding transportable features under the SoM assumption.
  • Evaluation vs. Search: The authors distinguish between Evaluation (verifying if a representation is trustworthy) and Search (finding the representation). This work focuses on providing mathematical rigor for the Evaluation step.

Results

In synthetic experiments, the authors compared three models: Empirical Risk Minimization (ERM), Invariance-based (INV), and their proposed Transportability-based (TR) method.

Comparison of loss across sample sizes for ERM, INV, and the proposed TR method.

As shown in the results, ERM performed poorly by absorbing spurious correlations that changed in the target domain. While INV performed better, it was limited in predictive power. The TR method significantly outperformed both as sample size increased, achieving the lowest cross-entropy loss on the target domain. This confirms that leveraging a full transportable representation—even one including non-causal elements—is superior to using simple invariant or causal subsets.

Limitations and Future Work

The theory relies heavily on the Stability of Mechanisms (SoM) assumption; if a target domain changes in a way not hinted at in the source data, guarantees cannot be made. Additionally, while the paper provides a framework for evaluation, it does not yet offer a deep-learning-ready algorithm for searching high-dimensional spaces like raw pixels. Future work aims to apply these transportability principles to algorithmic fairness and multi-calibrated scores.

Summary updated

BibTeX
@inproceedings{jalaldoust2024transportable,
  author = {Kasra Jalaldoust and Elias Bareinboim},
  title = {{Transportable Representations for Domain Generalization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=LCd32JhMwX},
}
Figure