Model-Based Domain Generalization

Alexander Robey, George J. Pappas, Hamed Hassani

Deep learning models are notoriously brittle. A model trained to identify tumors in one hospital might fail catastrophically at another simply because the lighting, equipment, or staining chemicals are slightly different. This is the core challenge of Domain Generalization (DG): how do we train a model on a few "training domains" so that it performs reliably on a completely unseen "test domain"?

In this paper, the authors introduce Model-Based Domain Generalization (MBDG). Their approach moves away from simple heuristics and instead uses generative modeling and optimization theory to force models to be truly invariant to environmental shifts. By framing this as a constrained optimization problem and solving its dual, the authors achieve massive performance boosts—up to 30%—over existing state-of-the-art methods.

The Problem of Domain Generalization

Most machine learning models assume that training data and test data come from the same distribution. In the real world, this is rarely true. Models often latch onto spurious correlations—features that are predictive in the training set but irrelevant to the actual task (e.g., a cow is usually on grass, so the model learns that "grass = cow"). When the model sees a cow on a beach, it fails.

Existing DG methods try to learn "invariant features," but recent large-scale studies have shown that they rarely outperform simple Empirical Risk Minimization (ERM)—essentially, just training on all available data—when ERM is properly tuned. The gap is a lack of a principled way to define and enforce what "invariance" actually means across domains.

An overview of the Model-Based Domain Generalization framework, illustrating how data from different hospitals represent distinct domains and how a generative model characterizes these variations.

The MBDG Approach

To address this, the authors propose a three-step framework that explicitly enforces invariance:

  1. Learn the Transformation ($G$): Instead of guessing how domains differ, they use a generative model (specifically MUNIT, a multimodal unsupervised image-to-image translation network). This model learns to "translate" an image from one domain's style to another (e.g., turning a "Photo" of a dog into a "Sketch" of a dog) without changing the underlying content.
  2. Define $G$-Invariance: They introduce a formal condition: a classifier should produce the same output for an image $x$ and its translated version $G(x, e)$. If the prediction changes when you change the "style" of the image, the model is relying on the wrong features.
  3. Primal-Dual Optimization: Enforcing this "hard" constraint is mathematically difficult. The authors use nonconvex duality theory to turn the constrained problem into a "saddle-point" problem. They introduce a dual variable ($\lambda$) that acts as a dynamic penalty. If the model's predictions start to vary across domains, $\lambda$ increases, forcing the model to prioritize invariance.

This differs from standard data augmentation because the penalty is mathematically tied to the optimization objective, ensuring the model actually converges to an invariant solution rather than just seeing more varied data.

Key Insights

  • The "Pull-Back" Principle: By using a generative model $G$, the authors "pull back" the complex problem of generalizing to unseen domains into a shared, underlying distribution.
  • Dynamic Penalties vs. Fixed Hyperparameters: Most DG methods use a fixed penalty weight for invariance. MBDG’s dual variable $\lambda$ is updated during training. If the model is "cheating" by using domain-specific features, the algorithm automatically cranks up the pressure to be invariant.
  • Covariate Shift Focus: The theory relies on the assumption that while the appearance of data changes, the meaning of the labels stays the same. This is a common and realistic assumption for many computer vision tasks.

Datasets and Results

The authors evaluated MBDG on several rigorous benchmarks, producing some of the most significant jumps in DG performance seen in recent years:

  • ColoredMNIST: A synthetic dataset where digit color is highly correlated with the label in training but changes in testing. MBDG achieved 68.5% average accuracy, nearly 30% higher than the Invariant Risk Minimization (IRM) baseline.
  • Camelyon17-WILDS: A medical dataset involving tumor identification across different hospitals. Accuracy jumped from 73.3% (ERM) to 94.8%—a massive improvement for a real-world medical application.
  • PACS: A standard DG benchmark featuring Photos, Art, Cartoons, and Sketches. MBDG set a new state-of-the-art, particularly on the "Sketch" domain, improving accuracy by over 10%.
  • FMoW-WILDS: Satellite imagery taken across different years and geographical regions.

Crucially, ablation studies showed that simply using the generative model for data augmentation wasn't enough; the primal-dual iteration was the "secret sauce" that forced the model to learn the right representations.

Limitations and Future Work

While powerful, MBDG has limitations. It is highly dependent on the quality of the generative model ($G$); if the translation network fails to capture the essential transformations between domains, the invariance constraint loses its meaning. Additionally, training a high-quality image-to-image translation model before the main classifier adds significant computational overhead. Finally, the framework is designed for "Covariate Shift" (style changes) and does not handle "Concept Shift," where the actual relationship between features and labels changes across domains.

Future work may involve learning the transformation model and the classifier simultaneously to reduce the two-step overhead and streamline the training process.

Summary updated

Abstract

Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribution data. Toward addressing this challenge, we consider the \emph{domain generalization} problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on a distinct and unseen test domain. We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to an infinite-dimensional constrained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization. Due to the inherent challenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory to develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap. Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees. In our experiments, we report improvements of up to 30% over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WILDS, FMoW-WILDS, and PACS.
BibTeX
@inproceedings{robey2021model,
  author = {Alexander Robey and George J. Pappas and Hamed Hassani},
  title = {{Model-Based Domain Generalization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=JOxB9h40A-1},
}
Figure