Model-Based Domain Generalization

Alexander Robey, George J. Pappas, Hamed Hassani

Abstract

Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribution data. Toward addressing this challenge, we consider the domain generalization problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on a distinct and unseen test domain. We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to an infinite-dimensional constrained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization. Due to the inherent challenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory to develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap. Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees. In our experiments, we report improvements of up to 30 percentage points over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WILDS, FMoW-WILDS, and PACS.

Introduction: The Brittleness of Deep Learning

Deep learning models are notoriously brittle when faced with distribution shifts. A model trained to identify cancerous cells at one hospital might fail catastrophically when deployed at another due to subtle differences in lighting, staining techniques, or equipment. This challenge is known as the Domain Generalization (DG) problem: training a model on a set of "training domains" such that it maintains performance on a completely unseen "test domain."

Figure 1: The domain generalization problem illustrated with histology images from different hospitals, representing distinct training domains.

Most current approaches struggle because they attempt to find "invariant features" without a clear model of what actually changes between environments. Recent studies have even shown that simple Empirical Risk Minimization (ERM)—standard training—often outperforms sophisticated DG algorithms when properly tuned. This research identifies a specific challenge called covariate shift, where the appearance of data changes across environments while the underlying relationship between the data and its label remains constant.

Methodology: Model-Based Domain Generalization (MBDG)

The proposed framework, Model-Based Domain Generalization (MBDG), addresses these shifts through a two-step process:

  1. Learning the Shift: The framework utilizes an unsupervised generative model (specifically MUNIT, a multimodal image-to-image translation network) to learn a domain transformation model $G(x, e)$. This model can take an image from one domain and "re-imagine" it in another, effectively simulating environmental variations.
  2. Enforcing Invariance: The classifier is trained with a strict constraint: its prediction for an image $x$ must be identical to its prediction for the transformed image $G(x, e)$. This is defined as $G$-invariance.
Figure 2: The generative process where an unobserved latent variable X is transformed into observed data across different environments e via the function G.

To make this mathematically tractable, the problem is reformulated as a semi-infinite constrained optimization problem. Since neural networks are difficult to optimize under hard constraints, the authors employ non-convex duality theory to create an unconstrained "dual" version of the problem. This is solved using a primal-dual iteration, where the model minimizes loss while a learned penalty weight (dual variable) enforces the invariance constraints.

Experimental Results

The researchers evaluated MBDG on several rigorous benchmarks, including ColoredMNIST, Camelyon17-WILDS (medical imaging), FMoW-WILDS (satellite imagery), and PACS (Art/Cartoon/Sketch). MBDG delivered some of the most significant improvements seen in recent DG literature:

  • Camelyon17-WILDS: Accuracy increased from 73.3% (ERM) to 94.8%, a massive leap for real-world medical diagnostics.
  • ColoredMNIST: In the most difficult setting where correlations are reversed, MBDG achieved 68.5% average accuracy, outperforming the Invariant Risk Minimization (IRM) baseline by nearly 30 percentage points.
  • PACS: The model achieved a new state-of-the-art average accuracy of 85.6%.

Conclusion

MBDG shifts the focus from internal feature invariance to output-level prediction invariance. By using generative models to simulate potential unseen environments and a principled primal-dual approach for optimization, it provides a robust solution to covariate shift. While the method is computationally more intensive than standard training and depends on the quality of the generative model $G$, its performance gains suggest a significant step forward for reliable AI deployment in changing environments.

Summary updated

Abstract

Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribution data. Toward addressing this challenge, we consider the domain generalization problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on a distinct and unseen test domain. We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to an infinite-dimensional constrained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization. Due to the inherent challenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory to develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap. Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees. In our experiments, we report improvements of up to 30 percentage points over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WILDS, FMoW-WILDS, and PACS.
BibTeX
@inproceedings{robey2021model,
  author = {Alexander Robey and George J. Pappas and Hamed Hassani},
  title = {{Model-Based Domain Generalization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=JOxB9h40A-1},
}
Figure