Introduction
Deep learning models often operate under the assumption that training (source) and test (target) data originate from the identical distribution. However, in real-world deployments, this assumption frequently fails—a self-driving car trained in sunny weather may struggle when suddenly driving in the snow. To tackle this Out-of-Distribution (OOD) generalization problem, existing methods typically attempt to learn "invariant features" across different source domains.
Despite these efforts, many domain generalization methods underperform basic Empirical Risk Minimization (ERM) in practice. This sharp gap between theory and practice raises a fundamental question: What kind of invariance actually guarantees that a good model on a source domain will perform well on a related target domain? This paper addresses that gap by proposing a new, computable criterion called transferability.
A New Definition of Transferability
Existing theories often focus solely on matching marginal feature distributions (using metrics like Wasserstein distance or MMD), ignoring the joint distribution of features and labels. The authors shift the focus to the excess risk of a predictor.
In this new framework, a source domain is considered transferable to a target domain if any near-optimal classifier on the source is also near-optimal on the target. The authors introduce quantifiable "transfer measures" that bound the target error, which can be estimated from finite samples using standard surrogate losses like cross-entropy.
Crucially, this proposed transfer measure is mathematically weaker than standard discrepancy measures like the Total Variation distance. This is a significant advantage: it demonstrates that two domains can be visually or statistically distinct yet remain highly transferable for a specific classification task, provided their decision boundaries align.
Adversarial Evaluation and Defense
To operationalize this theory, the authors introduce two algorithms for evaluating and improving transferability.
Evaluating Transferability (The Attack): To test if a model has genuinely learned transferable features, the authors designed an adversarial evaluation. It perturbs the learned classifier's parameters within a small radius ($\delta$) to find a state that maintains high accuracy on the source domain but fails catastrophically on the target domain.
Improving Transferability (The Defense): To mitigate this vulnerability, the authors propose a minimax optimization (adversarial training) algorithm. This method learns a feature embedding that minimizes the worst-case transferability gap among source domains while preserving high accuracy on the source tasks.
Experimental Results
The authors evaluated their methods across several domain generalization benchmarks, including RotatedMNIST, PACS, Office-Home, and the large-scale WILDS-FMoW dataset.
- Vulnerability of Existing Methods: The adversarial evaluation revealed a glaring weakness in current algorithms like ERM, DANN, and Mixup. For instance, on the PACS dataset using Mixup, slightly perturbing the classifier caused target accuracy to drop by more than 30%, while source accuracy barely changed. This confirms that the underlying features learned by these models are not inherently transferable. Algorithms that naturally learn more robust classifiers, such as Correlation Alignment (CORAL), demonstrated better transferability under this metric.
- Improved Generalization: The proposed adversarial defense algorithm achieved consistent improvements over state-of-the-art methods across all tested datasets.
- Tighter Theoretical Bounds: The authors demonstrated that their transfer measure provides a tighter theoretical bound on target error compared to traditional $\mathcal{H}$-divergence, particularly in datasets like PACS where label shift between domains is significant.
Limitations and Future Directions
While effective, the proposed defense algorithm requires a large number of inner maximization steps to compute the adversarial gap, making training significantly slower than standard methods—akin to the overhead seen in adversarial robustness training. It also requires moderate hyperparameter tuning for parameters like the perturbation radius ($\delta$) and the inner optimizer's learning rate.
For future work, the authors suggest exploring "certified robust feature embeddings." Drawing analogies to certified robust classifiers in the adversarial robustness literature, this direction could provide strict mathematical guarantees against distribution shifts in the real world.
Summary updated