Barycentric-Alignment and Reconstruction Loss Minimization for Domain Generalization

Boyang Lyu, Thuan Nguyen, Prakash Ishwar, Matthias Scheutz, Shuchin Aeron

TL;DR

This paper introduces a novel theoretical framework and algorithm for Domain Generalization (DG) by deriving a new, fully optimizable upper bound on the risk in unseen domains. This bound incorporates Wasserstein-2 barycenter for domain alignment and a reconstruction loss, leading to a practical algorithm (WBAE) that outperforms existing state-of-the-art DG methods.

What Problem Does This Paper Address?

Modern machine learning models often struggle to generalize to data from different distributions than what they were trained on. This is a critical issue in real-world applications where data sources can vary (e.g., medical images from different hospitals, sensor readings from different environments). Domain Generalization (DG) aims to train models that can perform well on unseen domains without explicit access to their data during training. Existing DG methods often rely on theoretical upper bounds of the classification risk, but these bounds contain terms that are difficult to optimize in practice due to their dependence on unknown optimal labeling functions. This paper addresses this gap by proposing a new upper bound that is free from such intractable terms, enabling more effective and practical DG solutions.

Methods

The core of this work lies in deriving a new theoretical upper bound for the risk in unseen domains. This bound is composed of four terms:

  1. Empirical risk on seen domains: Standard classification loss on the training data.
  2. Wasserstein-2 barycenter term: This novel term measures the discrepancy between the representation distributions of seen and unseen domains. It encourages domain alignment in the learned feature space.
  3. Reconstruction loss term: Another novel component, this term assesses how well the input data can be reconstructed from its representation. This encourages the learned representations to retain sufficient information about the original data.
  4. Combined risk term: This term is designed to be independent of the representation and labeling functions, making it optimizable.

Based on this new theoretical bound, the authors propose the Wasserstein Barycenter Auto-Encoder (WBAE) algorithm. WBAE is an auto-encoder-based model that jointly optimizes:

  • A cross-entropy loss for classification on seen domains.
  • A Wasserstein-2 barycenter loss to align domain representations.
  • A reconstruction loss to ensure the quality of learned representations.

The Wasserstein barycenter is approximated using an efficient Sinkhorn divergence method.

Datasets

The proposed WBAE algorithm was evaluated on four standard benchmark datasets for Domain Generalization:

  • PACS: Contains images from Art, Cartoons, Photos, and Sketches domains.
  • VLCS: Consists of images from VOC2007, LabelMe, Caltech, and PASCAL domains.
  • Office-Home: Includes Artistic, Clipart, Product, and Real-World domains.
  • TerraIncognita: Features four domains (L100, L38, L43, L46) of wild animal images.

Experiments were conducted under two settings: the DomainBed setting and the Stochastic Weight Averaging Densely (SWAD) setting.

Key Insights and Ideas

  • Bridging Theory and Practice: The paper successfully bridges the gap between theoretical DG bounds and practical algorithm design by introducing a new, fully optimizable upper bound. This is achieved by carefully reformulating the bound to remove intractable terms.
  • Representation Space Alignment: A key insight is that aligning domain distributions in the representation space (using Wasserstein-2 barycenter) is more theoretically sound and practically aligned than aligning them in the original data space, as done in some prior work.
  • Reconstruction as a DG Signal: The introduction of the reconstruction loss as a component of the DG upper bound is novel. It suggests that encouraging representations to be "invertible" (i.e., allowing reconstruction of the input) can improve generalization.
  • Unified Objective: The WBAE algorithm elegantly combines classification, domain alignment, and representation quality into a single, coherent optimization objective, directly motivated by the derived theoretical bound.

Key Results

  • State-of-the-Art Performance: WBAE consistently outperforms existing state-of-the-art DG algorithms across the four benchmark datasets.
  • Significant Improvements: On the PACS dataset, WBAE achieved an average improvement of 1.7-2.8 percentage points over theory-guided methods. It also showed notable gains on the Office-Home and TerraIncognita datasets.
  • Competitive with Heuristics: WBAE performs comparably or slightly better than the best-performing heuristic DG methods.
  • Ablation Study: An ablation study confirmed the importance of both the Wasserstein barycenter loss (Lbary) and the reconstruction loss (Lr). Removing Lbary led to a more significant performance drop (1.2-3.1%) compared to removing Lr (0.4-1.1%), highlighting Lbary's primary role in domain alignment.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that estimating the Wasserstein-2 barycenter can be computationally expensive, especially with larger batch sizes for more accurate estimations.
  • Future Work: To address the computational constraint, future work will focus on leveraging recently proposed large-scale barycenter and mapping estimators to enable barycenter calculation with a larger number of samples.

Summary updated

Abstract

This paper advances the theory and practice of Domain Generalization (DG) in machine learning. We consider the typical DG setting where the hypothesis is composed of a representation mapping followed by a labeling function. Within this setting, the majority of popular DG methods aim to jointly learn the representation and the labeling functions by minimizing a well-known upper bound for the classification risk in the unseen domain. In practice, however, methods based on this theoretical upper bound ignore a term that cannot be directly optimized due to its dual dependence on both the representation mapping and the unknown optimal labeling function in the unseen domain. To bridge this gap between theory and practice, we introduce a new upper bound that is free of terms having such dual dependence, resulting in a fully optimizable risk upper bound for the unseen domain. Our derivation leverages classical and recent transport inequalities that link optimal transport metrics with information-theoretic measures. Compared to previous bounds, our bound introduces two new terms: (i) the Wasserstein-2 barycenter term that aligns distributions between domains, and (ii) the reconstruction loss term that assesses the quality of representation in reconstructing the original data. Based on this new upper bound, we propose a novel DG algorithm named Wasserstein Barycenter Auto-Encoder (WBAE) that simultaneously minimizes the classification loss, the barycenter loss, and the reconstruction loss. Numerical results demonstrate that the proposed method outperforms current state-of-the-art DG algorithms on several datasets.
BibTeX
@inproceedings{lyu2023barycentric,
  author = {Boyang Lyu and Thuan Nguyen and Prakash Ishwar and Matthias Scheutz and Shuchin Aeron},
  title = {{Barycentric-Alignment and Reconstruction Loss Minimization for Domain Generalization}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=R2dBXce6P1},
}
Figure