Optimal Representations for Covariate Shift

Yangjun Ruan, Yann Dubois, Chris J. Maddison

Introduction

Machine learning models often experience a sharp drop in performance when deployed in the real world due to distribution shifts—where the "target" (test) data differs significantly from the "source" (training) data. A common goal in Domain Generalization (DG) is to learn representations that ignore domain-specific noise (e.g., the stylistic difference between a photo and a sketch) while retaining the core signal necessary for the task.

Despite many proposed DG methods, simple Empirical Risk Minimization (ERM) often remains a strong baseline. This paper addresses a fundamental gap: defining the minimal theoretical requirements for a representation to be perfectly robust to covariate shifts. Furthermore, it tackles the "No Free Lunch" problem in DG, proving that it is mathematically impossible to learn a representation that universally outperforms a constant predictor without some knowledge of the target distribution.

The Optimal Representation Theorem

The authors bridge the gap between theory and practice by identifying the "minimal sufficient" conditions for learning robust representations under covariate shift.

Theorem 1 (Optimal Representations): For a representation to be optimal and robust, it must satisfy two key conditions:

  1. Discriminative Power: There must exist a predictor capable of minimizing risk on both the source and target domains simultaneously.
  2. Support Match: The "support" (the set of possible values or boundaries) of the representation must be identical across all domains. Previous work often focused on matching the entire distribution shape, but this paper argues that matching the support is the fundamental requirement.
Optimal representations require both discriminative power and support matching.

Practical Implementation via Domain-Agnostic Augmentations

To enforce these theoretical conditions without requiring labeled target data, the authors propose using domain-agnostic augmentations. These augmentations preserve the underlying label while stripping away domain-specific information.

A prime example is image-to-text mapping. A photograph of a dog and a sketch of a dog belong to different visual domains, but both can map to the identical text description "A dog." In this way, text acts as a domain-agnostic augmentation.

Image-to-text mapping serves as a domain-agnostic augmentation, mapping different visual domains to the same representation.

To operationalize this, the authors introduce two variational objectives:

  • Contrastive Adversarial Domain (CAD) Bottleneck: A stable objective utilizing contrastive learning to make the domain indistinguishable from the representation, avoiding the training instability typical of standard adversarial networks.
  • Entropy (Ent) Bottleneck: A method that compresses the representation, forcing the model to discard non-essential domain information.

Key Insights and Results

This theoretical framework provides a compelling explanation for the remarkable robustness of vision-language models like CLIP. Because CLIP is trained on image-text pairs, and text naturally acts as a domain-agnostic augmentation, CLIP inherently learns the optimal representations described in Theorem 1.

By fine-tuning CLIP with the CAD bottleneck, the authors achieved state-of-the-art (SOTA) results across standard Domain Generalization benchmarks in the DomainBed suite (including PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet).

  • DomainNet: On this massive and difficult benchmark, the CLIP L + CAD method achieved 53.7% accuracy, obliterating the previous SOTA of 41.8%.
  • PACS: The method reached an impressive 94.9% accuracy.
  • Agnostic vs. Standard Augmentations: Controlled experiments demonstrated that domain-agnostic augmentations yielded a target log-likelihood of -0.8, whereas standard self-supervised augmentations (like SimCLR) performed poorly at -4.2.

Limitations and Future Work

While the theoretical and empirical results are strong, the paper notes a few limitations. The core theory assumes an "Idealized DG" setting with access to population risks, whereas real-world scenarios involve finite samples and estimation errors. Additionally, the proofs assume unconstrained predictors; restricting the hypothesis class (e.g., to linear predictors) might require adjusting the support match constraint. Finally, the focus is strictly on covariate shift, leaving "concept drift" (where the relationship between features and labels changes) for future work.

Summary updated

Abstract

Machine learning systems often experience a distribution shift between training and testing. In this paper, we introduce a simple variational objective whose optima are exactly the set of all representations on which risk minimizers are guaranteed to be robust to any distribution shift that preserves the Bayes predictor, e.g., covariate shifts. Our objective has two components. First, a representation must remain discriminative for the task, i.e., some predictor must be able to simultaneously minimize the source and target risk. Second, the representation's marginal support needs to be the same across source and target. We make this practical by designing self-supervised objectives that only use unlabelled data and augmentations to train robust representations. Our objectives give insights into the robustness of CLIP, and further improve CLIP's representations to achieve SOTA results on DomainBed.
BibTeX
@inproceedings{ruan2022optimal,
  author = {Yangjun Ruan and Yann Dubois and Chris J. Maddison},
  title = {{Optimal Representations for Covariate Shift}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=Rf58LPCwJj0},
}
Figure