Abstract
Out-of-distribution generalization is one of the key challenges when transferring a model from the lab to the real world. Existing efforts mostly focus on building invariant features among source and target domains. Based on invariant features, a high-performing classifier on source domains could hopefully behave equally well on a target domain. In other words, the invariant features are transferable. However, in practice, there are no perfectly transferable features, and some algorithms seem to learn "more transferable" features than others. How can we understand and quantify such transferability? In this paper, we formally define transferability that one can quantify and compute in domain generalization. We point out the difference and connection with common discrepancy measures between domains, such as total variation and Wasserstein distance. We then prove that our transferability can be estimated with enough samples and give a new upper bound for the target error based on our transferability. Empirically, we evaluate the transferability of the feature embeddings learned by existing algorithms for domain generalization. Surprisingly, we find that many algorithms are not quite learning transferable features, although few could still survive. In light of this, we propose a new algorithm for learning transferable features and test it over various benchmark datasets, including RotatedMNIST, PACS, Office-Home and WILDS-FMoW. Experimental results show that the proposed algorithm achieves consistent improvement over many state-of-the-art algorithms, corroborating our theoretical findings.
Introduction
The fundamental assumption of deep learning is that training and test data come from the same distribution. In the real world, this assumption is frequently violated—for instance, a self-driving car trained in sunny California may fail in a snowy Canadian winter. This challenge is known as Domain Generalization (DG).
Existing DG methods typically attempt to find "invariant features"—characteristics that remain consistent across all training domains. However, a persistent gap remains between theory and practice: simple Empirical Risk Minimization (ERM) often outperforms complex "invariant" algorithms. This paper argues that current methods focus too much on matching feature distributions while ignoring the excess risk. The central question is: How can we guarantee that a model that is "near-optimal" on training data will also be "near-optimal" on a completely new domain?
Quantifying Transferability
The authors propose a formal mathematical framework to quantify "transferability" based on excess risk. If a classifier is nearly as good as the best possible classifier on the source domain, it should ideally maintain that relative performance on the target domain.
Key theoretical insights include:
- Transfer Measures: New measures are derived to compute the difference in excess risks between domains. Unlike older metrics like $\mathcal{H}$-divergence, these account for the joint distribution of features and labels.
- Transferability vs. Similarity: The authors prove that domains do not need to be identical (low Total Variation distance) to be transferable. This explains why humans can transfer knowledge from a line drawing (Sketch) to a photograph (Photo) despite pixel-level differences.
- Tighter Bounds: The paper provides a new mathematical upper bound for target error that is tighter and more useful than classic bounds, especially in the presence of label shift.
The Transfer Algorithm
To improve DG, the authors develop a minimax optimization algorithm designed to learn transferable features:
- The "Attack" (Inner Loop): The algorithm searches for a classifier in the neighborhood of the current model that maximizes the performance gap between source domains.
- The "Defense" (Outer Loop): It updates the feature extractor to minimize both the average source error and this worst-case performance gap.
This approach functions like adversarial training, but instead of perturbing pixels, it "perturbs" the classifier's parameters to identify and eliminate non-transferable regions of the feature space.
Empirical Findings
The proposed approach was evaluated on four benchmarks: RotatedMNIST, PACS, Office-Home, and the large-scale satellite imagery dataset WILDS-FMoW.
The most striking finding is the "Surviving vs. Transferable" Paradox. Many popular DG algorithms (like DANN or Mixup) do not actually learn transferable features. They "survive" on the target domain by luck. As shown in the empirical results above, when the model parameters are slightly perturbed (increasing $\delta$), target accuracy crashes while source accuracy remains stable. For the PACS dataset, a Mixup-trained model saw target accuracy drop by over 30% under small perturbations.
Results and Conclusion
The proposed "Transfer" algorithm consistently improved upon or matched state-of-the-art algorithms like CORAL and Spectral Decoupling (SD) across all datasets. On the WILDS-FMoW dataset, it demonstrated superior stability compared to GroupDRO and IRM.
While the algorithm is computationally expensive due to the inner maximization steps and sensitive to the hyperparameter $\delta$, it provides a path toward "certified" robust feature embeddings. By quantifying transferability, this work offers a more rigorous foundation for developing models that can truly generalize to the real world.
Summary updated