Domain Generalization using Causal Matching

Divyat Mahajan, Shruti Tople, Amit Sharma

Abstract: In the domain generalization literature, a common objective is to learn representations independent of the domain after conditioning on the class label. We show that this objective is not sufficient: there exist counter-examples where a model fails to generalize to unseen domains even after satisfying class-conditional domain invariance. We formalize this observation through a structural causal model and show the importance of modeling within-class variations for generalization. Specifically, classes contain objects that characterize specific causal features, and domains can be interpreted as interventions on these objects that change non-causal features. We highlight an alternative condition: inputs across domains should have the same representation if they are derived from the same object. Based on this objective, we propose matching-based algorithms when base objects are observed (e.g., through data augmentation) and approximate the objective when objects are not observed (MatchDG). Our simple matching-based algorithms are competitive to prior work on out-of-domain accuracy for rotated MNIST, Fashion-MNIST, PACS, and Chest-Xray datasets. Our method MatchDG also recovers ground-truth object matches: on MNIST and Fashion-MNIST, top-10 matches from MatchDG have over 50% overlap with ground-truth matches.

What Problem Does This Paper Address?

Domain Generalization (DG) is the challenge of training a machine learning model on several different data distributions (domains) so that it performs well on a completely new, unseen domain. For example, a model trained to recognize digits on white backgrounds should ideally work on digits with colored or rotated backgrounds.

The prevailing wisdom in DG has been to learn representations that are class-conditional domain invariant. In plain English: if you know an image is a "2," its internal representation shouldn't tell you which domain it came from.

However, this paper identifies a major gap: within a single class (like the digit "2"), there is still massive variation. Different people have different handwriting styles (which the authors call "objects"). If the distribution of these styles changes across domains, standard methods fail. They might latch onto "spurious" features that happen to satisfy the invariance constraint in training but disappear in the test set.

A counter-example showing how a predictor satisfying class-conditional domain invariance can fail to generalize.

Methods

The authors introduce a Structural Causal Model (SCM) to formalize the data-generation process. They posit that an input is a mix of "stable" (causal) features and "domain-dependent" (non-causal) features. An ideal classifier should only care about the causal ones.

Structural Causal Models (SCM) for (a) image classification and (b) the general case, highlighting causal (Xc) and non-causal (Xa) features.

To find these causal features, they propose Causal Matching. The core idea is: if two images from different domains represent the same "object" (e.g., the same person's handwriting), they should have the same representation. Since we rarely know which images represent the same "object" in real-world data, they developed MatchDG, a two-phase approach:

  1. Phase I (Inference): Use iterative contrastive learning to find "matches." The model starts with random same-class pairs and gradually learns a representation where images of the same "object" (sharing causal features) are closer together than images of different objects, even within the same class.
  2. Phase II (Classification): Train the actual classifier using the matches found in Phase I as a regularizer. This forces the network to ignore domain-specific noise and focus on the stable features that define the object.

They also introduce MDGHybrid, which combines these inferred matches with known matches generated through data augmentation (like rotating an image).

Key Insights and Results

The authors evaluated their approach on four diverse benchmarks: Rotated MNIST & Fashion-MNIST, PACS, and a real-world Chest-Xray dataset.

  • Invariance isn't enough: The paper provides a theoretical counter-example showing that a model can be perfectly domain-invariant during training and still fail on a test set if it hasn't captured the true causal features.
  • Superior Accuracy: On Rotated MNIST and Fashion-MNIST, MatchDG outperformed state-of-the-art baselines like IRM (Invariant Risk Minimization). For example, on the hardest Fashion-MNIST setup, it achieved 33.0% accuracy vs. the next best at 26.9%.
  • Real-world Robustness: On the Chest-Xray dataset, MDGHybrid achieved 64.3% accuracy on the RSNA target domain, a significant jump over standard Empirical Risk Minimization (ERM) at 55.1%.
  • Recovering Ground Truth: Remarkably, even though MatchDG isn't told which images are "matches," its top-10 predicted matches had over 50% overlap with the actual ground-truth objects on MNIST datasets.

Conclusion

By shifting the focus from "classes" to "objects" (within-class variations), the model gains a much finer-grained signal for what features are truly stable. The authors demonstrate that contrastive learning can be repurposed to uncover the underlying causal structure of a dataset without needing explicit causal labels, leading to models that generalize significantly better to unseen environments.

Summary updated

Abstract

In the domain generalization literature, a common objective is to learn representations independent of the domain after conditioning on the class label. We show that this objective is not sufficient: there exist counter-examples where a model fails to generalize to unseen domains even after satisfying class-conditional domain invariance. We formalize this observation through a structural causal model and show the importance of modeling within-class variations for generalization. Specifically, classes contain objects that characterize specific causal features, and domains can be interpreted as interventions on these objects that change non-causal features. We highlight an alternative condition: inputs across domains should have the same representation if they are derived from the same object. Based on this objective, we propose matching-based algorithms when base objects are observed (e.g., through data augmentation) and approximate the objective when objects are not observed (MatchDG). Our simple matching-based algorithms are competitive to prior work on out-of-domain accuracy for rotated MNIST, Fashion-MNIST, PACS, and Chest-Xray datasets. Our method MatchDG also recovers ground-truth object matches: on MNIST and Fashion-MNIST, top-10 matches from MatchDG have over 50% overlap with ground-truth matches.
BibTeX
@inproceedings{mahajan2021domain,
  author = {Divyat Mahajan and Shruti Tople and Amit Sharma},
  title = {{Domain Generalization using Causal Matching}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=pkvkFrQdzgW},
}
Figure