Abstract
This paper advances the theory and practice of Domain Generalization (DG) by addressing a critical gap in existing risk upper bounds. Traditional DG methods often minimize a theoretical bound that includes a term dependent on both the representation mapping and the unknown optimal labeling function in unseen domains, making it impossible to optimize directly. We introduce a new, fully optimizable upper bound leveraging Wasserstein-2 barycenters and information-theoretic measures. This theoretical framework leads to the development of the Wasserstein Barycenter Auto-Encoder (WBAE), an algorithm that aligns distributions toward a central barycenter while preserving essential data features through reconstruction. Experimental results across standard benchmarks demonstrate that WBAE consistently outperforms state-of-the-art DG methods.
Introduction
In modern machine learning, the assumption that training and testing data are independent and identically distributed (i.i.d.) rarely holds in real-world deployments. For instance, a diagnostic model trained on medical imaging from one hospital may fail when applied to another due to differences in equipment, lighting, or patient demographics. This challenge is known as Domain Generalization (DG): the goal is to learn a model from multiple "seen" source domains that can generalize effectively to an "unseen" target domain.
Most DG approaches attempt to learn a domain-invariant representation mapping followed by a classifier. However, the theoretical foundations for these methods often rely on risk bounds that contain unobservable terms. This paper bridges the gap between theory and practice by deriving a bound where every term is actionable and optimizable.
The Theoretical Gap in Domain Generalization
Seminal work in DG theory suggests that the risk in an unseen domain is bounded by the source domain risk, a domain discrepancy term, and a "combined risk" term. Historically, researchers have struggled with the combined risk term because it depends on the unknown optimal labels of the unseen domain. Consequently, it was often ignored or assumed to be a constant.
This paper argues that ignoring this term is a mistake, as it actually depends on the model's representation mapping. By leveraging classical and recent transport inequalities, the authors derive a new bound that replaces this problematic term with two optimizable components: a Wasserstein-2 barycenter alignment and a reconstruction loss. This ensures that the representation mapping remains nearly invertible, preventing the loss of task-relevant information.
The WBAE Framework
Based on the new upper bound, the authors propose the Wasserstein Barycenter Auto-Encoder (WBAE). The algorithm minimizes a composite objective function consisting of three primary losses:
- Classification Loss: A standard cross-entropy loss applied to the seen domains to ensure the model learns accurate labeling.
- Barycenter Alignment Loss: Rather than performing computationally expensive pairwise alignment between all domains, WBAE aligns each seen domain toward a central "average" distribution—the Wasserstein-2 barycenter—in the feature space. This encourages the model to extract features that are invariant across all environments.
- Reconstruction Loss: To satisfy the theoretical requirement of manifold-restricted invertibility, the model includes a decoder that reconstructs the original input from the latent representation. This prevents "feature collapse," where the model might otherwise discard useful information to achieve domain invariance.
Experimental Results
The WBAE algorithm was evaluated on four major DG benchmarks: PACS, VLCS, Office-Home, and TerraIncognita. These datasets cover a wide range of shifts, from artistic styles to geographic camera trap variations.
Performance Highlights
- Standard Benchmarking: Under the standard DomainBed evaluation protocol, WBAE achieved an average accuracy of 70.7%, outperforming other theory-guided methods like IRM, DANN, and VREx.
- State-of-the-Art Integration: When combined with Stochastic Weight Averaging Densely (SWAD), WBAE reached an average accuracy of 72.8%, setting a new performance bar for these benchmarks.
- Ablation Studies: The authors confirmed the necessity of their theoretical components. Removing the barycenter loss resulted in a performance drop of up to 3.1%, while removing the reconstruction loss led to a decrease of up to 1.1%.
Key Insights
- Efficiency through Barycenters: For $S$ domains, aligning to a barycenter requires only $S$ Wasserstein distance calculations, compared to the $S(S-1)/2$ calculations required by pairwise alignment methods.
- Feature Space Alignment: Aligning distributions in the representation space rather than the raw input space is more computationally feasible and aligns better with the hierarchical nature of deep learning.
- The Role of Invertibility: The inclusion of an auto-encoder is not merely a heuristic; the theory demonstrates that maintaining a degree of invertibility is essential to control the risk bound in unseen domains.
Conclusion and Future Work
The WBAE framework provides a rigorous theoretical foundation for DG while delivering superior empirical performance. While the calculation of Wasserstein-2 barycenters introduces some computational complexity, the authors suggest that future work involving large-scale barycenter estimators could further scale the method. By ensuring that representation mappings are both domain-invariant and information-preserving, WBAE offers a robust path forward for reliable machine learning in changing environments.
Summary updated