Abstract
Machine learning systems often experience a distribution shift between training and testing. In this paper, we introduce a simple variational objective whose optima are exactly the set of all representations on which risk minimizers are guaranteed to be robust to any distribution shift that preserves the Bayes predictor, e.g., covariate shifts. Our objective has two components. First, a representation must remain discriminative for the task, i.e., some predictor must be able to simultaneously minimize the source and target risk. Second, the representation's marginal support needs to be the same across source and target. We make this practical by designing self-supervised objectives that only use unlabelled data and augmentations to train robust representations. Our objectives give insights into the robustness of CLIP, and further improve CLIP's representations to achieve SOTA results on DomainBed.
The Challenge of Domain Generalization
Machine learning models notoriously struggle with domain generalization (DG)—performing well on a target (test) distribution that differs from the source (train) distribution. A common strategy to combat this covariate shift is to learn "robust representations" that ignore domain-specific noise.
However, in practice, most representation learning methods for DG fail to consistently outperform simple Empirical Risk Minimization (ERM). Furthermore, the field has lacked a rigorous theoretical understanding of the minimal requirements for perfect domain generalization. The authors tackle this gap by asking: What exactly makes a representation optimal for DG, and why do current methods fall short?
Theoretical Foundations
The authors first tackle the problem theoretically in an Idealized Domain Generalization (IDG) setting (assuming access to population distributions). They mathematically characterize the necessary and sufficient conditions for optimal representations.
For a representation to be optimal under covariate shift, it must satisfy two criteria:
- Task-Discriminative: It must retain enough information so that some predictor can simultaneously minimize risk on both source and target domains.
- Support Matching: The marginal distribution of the representation must have the same support across both source and target domains.
The authors prove a "No Free Lunch" theorem for DG: without some knowledge of the target domain, no algorithm can uniformly outperform a simple constant predictor. This explains why many existing methods fail to beat ERM—they lack necessary target information.
Practical Implementation with SSL
To translate this theory into practice, the authors introduce a framework relying on Self-Supervised Learning (SSL) and domain-agnostic augmentations. Standard image augmentations (like color jittering) often retain domain-specific information. However, mapping an image to a text description (as in CLIP) preserves semantic task information while stripping away visual style.
To enforce the theoretical requirement of matching supports, the authors introduce two practical "domain bottlenecks":
- Contrastive Adversarial Domain (CAD) Bottleneck: Minimizes the mutual information between the representation and the domain using a contrastive variational bound, avoiding the instability of standard adversarial training (DANN).
- Entropy (Ent) Bottleneck: Minimizes the overall entropy of the representation to squeeze out excess information, forcing representations to match across domains.
Experimental Results
The proposed methods were evaluated on the DomainBed suite and natural distribution shift datasets. By fine-tuning a pretrained CLIP model (ViT-B/32) with the CAD bottleneck, the authors achieved significant state-of-the-art (SOTA) improvements:
- PACS: 94.9% (vs. previous SOTA 87.2%)
- OfficeHome: 80.0% (vs. previous SOTA 68.4%)
Furthermore, fine-tuning CLIP on the LAION-400M dataset using the Entropy bottleneck improved average robustness across seven natural distribution shift datasets (from 50.0% to 51.3%) compared to standard fine-tuning.
Conclusion
This work provides a rigorous theoretical foundation for domain generalization, identifying task-discriminative power and support matching as the twin pillars of robustness. By leveraging domain-agnostic augmentations (like image-text pairs) and novel variational bottlenecks, the authors demonstrate that theoretically grounded representation learning can lead to substantial practical gains in model robustness.
Summary updated