Deep learning models are notoriously brittle when deployed in environments that differ from their training data. To combat this "domain shift," researchers have developed an array of specialized domain generalization (DG) algorithms. These methods leverage complex loss functions, domain-alignment techniques, and causal invariance principles to force models to ignore domain-specific shortcuts.
However, rigorous empirical evaluations have revealed a striking paradox: standard Empirical Risk Minimization (ERM)—the simple baseline of pooling all available source data and training a standard model without any domain-specific modifications—consistently matches or outperforms these complex alternatives. Early signs of this phenomenon appeared when Da Li et al. demonstrated in Deeper, Broader and Artier Domain Generalization that simple data-pooling baselines frequently beat specialized algorithms on traditional benchmarks. This observation was later solidified by large-scale, controlled studies. For instance, Ramakrishna Vedantam et al. confirmed in An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers that standard ERM routinely equals or exceeds specialized DG methods, while Olivia Wiles et al. reached a similar conclusion in A Fine-Grained Analysis on Distribution Shift after training over 85,000 models. To understand why simple training is so resilient, we must look at the mathematical and information-theoretic foundations of the problem.
The Theoretical Optimality of ERM
The persistent success of ERM is not an accidental artifact of hyperparameter tuning; it is rooted in fundamental optimization theory. When a model must generalize to an unknown target domain, it faces an extrapolation problem. In An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization, Elan Rosenfeld et al. proved that ERM serves as a minimax-optimal strategy for both interpolation and extrapolation tasks. Without strong, restrictive structural assumptions about the unseen domains, minimizing the average risk across all observed environments is mathematically the safest strategy to minimize the worst-case error on a new environment.
Furthermore, specialized algorithms often assume that finding a domain-invariant representation requires complex, multi-objective optimization. However, Xin Zou et al. showed in Generalization Bounds for Out-of-distribution Generalization that if a single hypothesis within the model class is optimal for both the source and target domains, ERM achieves nearly optimal error rates. In other words, when a universal solution exists across environments, standard empirical risk minimization naturally converges to it, rendering the added complexity of domain-specific alignment objectives redundant.
The Pitfalls of Enforced Invariance
Many specialized algorithms fail because they try to explicitly enforce invariance or isolate purely causal features. The intuition behind these methods is noble: if a model only relies on features that have an invariant causal relationship with the label across training domains, it should generalize perfectly to a new domain.
In practice, this approach introduces severe vulnerabilities. As noted by Alexander Robey et al. in Model-Based Domain Generalization, existing methods lack a principled way to define and enforce invariance on complex data, causing models to miss universal features. More importantly, forcing a model to be strictly invariant often strips away valuable predictive power. In a large-scale empirical study on tabular datasets, Moritz Hardt et al. found in Predictors from causal features do not generalize better to new domains that "kitchen-sink" ERM models using all available features consistently outperform restricted causal predictors in both in-domain and out-of-distribution accuracy.
This finding was echoed by Vivian Y. Nastl et al. in Do causal predictors generalize better to new domains?, who observed that invariant learning methods often yield overly conservative models that sacrifice significant accuracy for marginal stability gains. Real-world data shifts are rarely clean; as Abbavaram Gowtham Reddy et al. argue in When Shift Happens - Confounding Is to Blame, the causal and invariance assumptions underlying specialized robustness algorithms frequently break down in the presence of complex real-world confounding, leaving standard ERM as the more robust and practical choice.
Real-World Benchmarks and the Data Bottleneck
The superiority of ERM is consistently visible across diverse, real-world benchmarks that simulate actual deployment conditions rather than synthetic shifts. In the WILDS benchmark introduced by Pang Wei Koh et al. (WILDS: A Benchmark of in-the-Wild Distribution Shifts), specialized robustness algorithms repeatedly failed to beat standard ERM on complex, natural variations. Similar trends appear under temporal shifts in the Wild-Time benchmark by Huaxiu Yao et al. (Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time), in time-series data evaluated by Jean-Christophe Gagnon-Audet et al. (WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series), and even in decentralized settings evaluated by Ruqi Bai et al. (Benchmarking Algorithms for Federated Domain Generalization).
Why do these advanced algorithms stall across such varied domains? Yimu Wang et al. provided a compelling information-theoretic explanation in Lost Domain Generalization Is a Natural Consequence of Lack of Training Domains. They proved that achieving low error on unseen target domains requires a number of training domains that scales polynomially with the desired accuracy. When a benchmark provides only a few training environments, the mediocre performance of specialized algorithms is not a failure of algorithmic design, but a fundamental data constraint. ERM succeeds because it maximizes the utility of the available data without introducing brittle, over-engineered constraints that collapse under data scarcity.
Unleashing ERM through Training Hygiene
If standard empirical risk minimization is fundamentally robust, then the key to closing the out-of-distribution performance gap lies not in inventing complex new algorithms, but in refining basic training hygiene.
This insight forms the basis of ERM++, an improved baseline proposed by Piotr Teterwak et al. in ERM++: An Improved Baseline for Domain Generalization. The authors demonstrated that by simply optimizing fundamental training components—such as data utilization, network initialization, and standard regularization—a properly tuned ERM baseline can outperform state-of-the-art domain generalization algorithms by over 15% without adding any architectural complexity.
Similarly, Boyan Gao et al. showed in Loss Function Learning for Domain Generalization by Implicit Gradient that replacing standard cross-entropy with a well-tuned loss function within a basic training pipeline allows simple models to outperform complex algorithmic machinery. This perspective shifts the focus of the field: instead of building intricate architectures to force domain invariance, researchers can achieve robust generalization by prioritizing data quality, model selection, and fundamental optimization hygiene.
Key Foundational Work
An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers (2021) — contributed 2 concept elements to this direction.
In Search of Lost Domain Generalization (2021) — contributed 2 concept elements to this direction.
Optimal Representations for Covariate Shifts (2021) — contributed 1 concept elements to this direction.