Joint covariate-alignment and concept-alignment: a framework for domain generalization

Thuan Nguyen, Boyang Lyu, Prakash Ishwar, Matthias Scheutz, Shuchin Aeron

Introduction

Machine learning models frequently encounter performance degradation when deployed in environments where the data distribution differs from the training set—a phenomenon known as distribution shift. In the field of Domain Generalization (DG), the objective is to train a model on multiple "seen" domains such that it generalizes effectively to a completely "unseen" target domain.

Traditional DG approaches typically address one of two primary types of shift:

  1. Covariate Shift: Changes in the marginal distribution of the input data (e.g., training on photos of sunny days but testing on rainy days).
  2. Concept Shift: Changes in the relationship between the input data and the labels (e.g., a feature that predicts a label in one domain becomes irrelevant or misleading in another).

This paper argues that focusing on either shift in isolation is theoretically insufficient. The authors propose a unified framework that jointly minimizes both covariate and concept shifts, grounded in a novel theoretical upper bound for risk on unseen domains.

Theoretical Foundation: A New Upper Bound

The core contribution of this work is the derivation of a new upper bound for prediction risk in an unseen domain. Unlike previous theoretical frameworks—such as the seminal Ben-David bound, which often treated the discrepancy between domains as a fixed constant—this new formulation identifies the domain mismatch as a variable that can be minimized through feature representation learning.

The authors demonstrate that the total risk on an unseen domain is bounded by three components:

  • Source Risk: The model's performance on the training (seen) domains.
  • Covariate-Alignment Term: A measure of the divergence between feature distributions across domains.
  • Concept-Alignment Term: A measure of the stability of the relationship between features and labels across domains.

By minimizing this joint bound, the model is forced to learn representations that are both distributionally similar and conceptually invariant. This provides a strong theoretical justification for why Invariant Risk Minimization (IRM) and its variants are necessary components of a larger DG strategy.

The Proposed Framework

To implement this theoretical bound, the authors introduce a modular framework that combines established alignment techniques. The framework is flexible, allowing for various combinations of covariate and concept alignment modules. In this study, the authors focus on two specific implementations: MMD-CEM and CORAL-CEM.

Covariate Alignment

The framework utilizes either Maximum Mean Discrepancy (MMD) or Correlation Alignment (CORAL). These methods aim to align the statistical moments of the feature distributions across different domains, ensuring the model does not overfit to domain-specific characteristics.

Concept Alignment

For concept alignment, the authors employ Conditional Entropy Invariant Risk Minimization (CEM). This approach seeks to identify features that maintain a stable, invariant relationship with the labels, regardless of the domain, effectively "anchoring" the model against spurious correlations.

Experimental Evaluation

The authors evaluated the proposed algorithms on two rigorous synthetic benchmarks designed to stress-test distribution shift: Colored MNIST (CMNIST) and Covariate-Shift-CMNIST (CS-CMNIST).

Results on CS-CMNIST

CS-CMNIST is particularly challenging as it incorporates both spurious color correlations (concept shift) and variations in the distribution of those colors (covariate shift). In this setting, the joint alignment approach demonstrated a clear advantage:

  • MMD-CEM achieved 90.7% accuracy.
  • CORAL-CEM reached 89.9% accuracy.

These results represent a significant improvement (nearly 4%) over single-focus methods like CEM (86.1%) and a dramatic leap over standard Empirical Risk Minimization (60.3%).

Results on CMNIST

On the standard CMNIST benchmark, which is primarily dominated by concept shift, the proposed methods performed comparably to existing state-of-the-art techniques (approximately 52.5%). The authors note that because this dataset lacks significant covariate shift, the addition of covariate alignment provides less of a performance boost than in the CS-CMNIST setting.

Discussion and Future Work

The findings underscore a critical insight: joint optimization is mandatory for robust generalization. Aligning only the features may lead to models that fail when the underlying "concept" changes, while aligning only the concepts may lead to overfitting on the training distribution.

However, the framework does introduce challenges, notably the sensitivity to hyperparameters ($\alpha$ and $\beta$) used to balance the alignment terms. Future research is needed to automate the selection of these parameters and to validate the framework on large-scale, real-world datasets such as those found in the Wilds or DomainNet benchmarks.

Conclusion

By bridging the gap between covariate and concept alignment, this framework provides a more comprehensive approach to Domain Generalization. The theoretical and empirical evidence suggests that anchoring models with both distributional and conceptual stability is the key to navigating the complexities of unseen data.

Summary updated

Abstract

In this paper, we propose a novel domain generalization (DG) framework based on a new upper bound to the risk on the unseen domain. Particularly, our framework proposes to jointly minimize both the covariate-shift as well as the concept-shift between the seen domains for a better performance on the unseen domain. While the proposed approach can be implemented via an arbitrary combination of covariate-alignment and concept-alignment modules, in this work we use well-established approaches for distributional alignment namely, Maximum Mean Discrepancy (MMD) and covariance Alignment (CORAL), and use an Invariant Risk Minimization (IRM)-based approach for concept alignment. Our numerical results show that the proposed methods perform as well as or better than the state-of-the-art for domain generalization on several data sets.
BibTeX
@inproceedings{nguyen2022joint,
  author = {Thuan Nguyen and Boyang Lyu and Prakash Ishwar and Matthias Scheutz and Shuchin Aeron},
  title = {{Joint covariate-alignment and concept-alignment: a framework for domain generalization}},
  booktitle = {MLSP},
  year = {2022},
  url = {https://openreview.net/forum?id=eRjWaQchWh},
}
Figure