A principled approach to model validation in domain generalization

Boyang Lyu, Thuan Nguyen, Matthias Scheutz, Prakash Ishwar, Shuchin Aeron

Abstract

Domain generalization (DG) aims to learn a model with good generalization ability, ensuring it performs well not only on several seen domains but also on unseen domains with different data distributions. State-of-the-art DG methods typically train a representation function followed by a classifier jointly to minimize both the classification risk and the domain discrepancy. However, when it comes to model selection, most of these methods rely on traditional validation routines that select models solely based on the lowest classification risk on the validation set. In this paper, we theoretically demonstrate a trade-off between minimizing classification risk and mitigating domain discrepancy, proving it is impossible to achieve the minimum of these two objectives simultaneously. Motivated by this theoretical result, we propose a novel model selection method suggesting that the validation process should account for both the classification risk and the domain discrepancy. We validate the effectiveness of the proposed method by numerical results on several domain generalization datasets.

Introduction

In standard machine learning, we assume that training and test data are independent and identically distributed (i.i.d.). Domain Generalization (DG) challenges this assumption by requiring models to perform on "unseen" domains that differ from the "seen" training distributions. For example, a model trained on photos and sketches of objects must generalize to oil paintings of those same objects.

While modern DG algorithms focus on learning "domain-invariant" features during training, a critical bottleneck remains: model selection. Most practitioners still select the final model based on the highest accuracy on a validation set composed of seen domains. This paper argues that this traditional approach is fundamentally flawed, as over-optimizing for accuracy on seen domains can sacrifice the invariance necessary for true generalization.

The Theoretical Trade-off

The core contribution of this research is the mathematical proof of a fundamental trade-off between classification risk and domain discrepancy. Drawing from Rate-Distortion theory, the authors demonstrate a convex, non-increasing relationship between these two objectives.

This proof implies that it is impossible to simultaneously minimize both to zero. Consequently, a model that achieves the absolute lowest error on seen validation data is likely to have high domain discrepancy, making it less robust when facing unseen data distributions. This theoretical insight suggests that the "best" model for generalization is rarely the one with the highest validation accuracy.

Proposed Methodology

To address this trade-off, the authors propose a shift from accuracy-only validation to a multi-objective validation loss ($L_{Val}$).

1. The Combined Validation Metric

The proposed metric balances classification performance with domain invariance:

$$L_{Val} = \beta(1 - \alpha)L_{Classification} + \alpha L_{Domain-Discrepancy}$$

  • $\alpha$: A hyperparameter (heuristically set to 0.2) that balances the importance of accuracy versus invariance.
  • $\beta$: A scaling factor to normalize the two terms if they differ in magnitude.
  • MMD (Maximum Mean Discrepancy): This is used to quantify the distance between feature distributions across different seen domains.

2. The Percentile Filter

Guided by the theory that extreme points are suboptimal, the authors introduce a "Percentile Filter." Instead of considering all models, they only evaluate those within the 5th to 50th percentile of validation error. This prevents the selection of models that have "cheated" by overfitting to the specific nuances of the seen domains (the 0-5th percentile).

Experimental Evaluation

The method was evaluated using the DomainBed benchmark across three diverse datasets:

  • Colored-MNIST (C-MNIST): A synthetic dataset where color is correlated with labels in training but reversed in testing. This serves as a rigorous test for whether a model has learned shapes or just spurious color correlations.
  • PACS: A vision benchmark featuring Photos, Art, Cartoons, and Sketches.
  • VLCS: A collection of real-world image datasets (VOC, LabelMe, Caltech, Sun) with natural distribution shifts.

Key Results

The proposed validation method was tested against 12 different DG algorithms, including IRM, CORAL, and DANN.

  • Significant Gains in C-MNIST: The VREx algorithm saw its accuracy jump from 10.2% to 15.6% simply by changing the validation strategy.
  • Improved Consistency: In the VLCS dataset, the new method selected superior models for 8 out of 12 algorithms, with accuracy gains up to 3.2%. In PACS, the method outperformed traditional validation in 7 out of 12 cases.
  • Robustness: Across all benchmarks, the combined metric was more consistent in selecting models that performed well on truly unseen test data compared to the standard accuracy-only approach.

Conclusion and Future Work

This research highlights that how we choose a model is just as important as how we train it. By incorporating domain discrepancy into the validation process, we can better identify models capable of true generalization.

However, some challenges remain. The hyperparameters $\alpha$ and $\beta$ are currently set heuristically, and finding an automated way to tune these for specific datasets is a priority for future work. Additionally, while MMD was used here, exploring other measures of domain discrepancy may further refine the validation process for different types of data shifts.

Summary updated

Abstract

Domain generalization aims to learn a model with good generalization ability, that is, the learned model should not only perform well on several seen domains but also on unseen domains with different data distributions. State-of-the-art domain generalization methods typically train a representation function followed by a classifier jointly to minimize both the classification risk and the domain discrepancy. However, when it comes to model selection, most of these methods rely on traditional validation routines that select models solely based on the lowest classification risk on the validation set. In this paper, we theoretically demonstrate a trade-off between minimizing classification risk and mitigating domain discrepancy, i.e., it is impossible to achieve the minimum of these two objectives simultaneously. Motivated by this theoretical result, we propose a novel model selection method suggesting that the validation process should account for both the classification risk and the domain discrepancy. We validate the effectiveness of the proposed method by numerical results on several domain generalization datasets.
BibTeX
@inproceedings{lyu2023principled,
  author = {Boyang Lyu and Thuan Nguyen and Matthias Scheutz and Prakash Ishwar and Shuchin Aeron},
  title = {{A principled approach to model validation in domain generalization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=kgHqF9pHgy},
}
Figure