Quantifying and Improving Transferability in Domain Generalization

Guojun Zhang, Han Zhao, Yaoliang Yu, Pascal Poupart

Introduction

Deep learning models often operate under the assumption that training (source) and test (target) data originate from the identical distribution. However, in real-world deployments, this assumption frequently fails—a self-driving car trained in sunny weather may struggle when suddenly driving in the snow. To tackle this Out-of-Distribution (OOD) generalization problem, existing methods typically attempt to learn "invariant features" across different source domains.

Despite these efforts, many domain generalization methods underperform basic Empirical Risk Minimization (ERM) in practice. This sharp gap between theory and practice raises a fundamental question: What kind of invariance actually guarantees that a good model on a source domain will perform well on a related target domain? This paper addresses that gap by proposing a new, computable criterion called transferability.

A New Definition of Transferability

Existing theories often focus solely on matching marginal feature distributions (using metrics like Wasserstein distance or MMD), ignoring the joint distribution of features and labels. The authors shift the focus to the excess risk of a predictor.

In this new framework, a source domain is considered transferable to a target domain if any near-optimal classifier on the source is also near-optimal on the target. The authors introduce quantifiable "transfer measures" that bound the target error, which can be estimated from finite samples using standard surrogate losses like cross-entropy.

Illustration of two domains that are dissimilar in feature space but share a perfectly transferable decision boundary.

Crucially, this proposed transfer measure is mathematically weaker than standard discrepancy measures like the Total Variation distance. This is a significant advantage: it demonstrates that two domains can be visually or statistically distinct yet remain highly transferable for a specific classification task, provided their decision boundaries align.

Adversarial Evaluation and Defense

To operationalize this theory, the authors introduce two algorithms for evaluating and improving transferability.

Evaluating Transferability (The Attack): To test if a model has genuinely learned transferable features, the authors designed an adversarial evaluation. It perturbs the learned classifier's parameters within a small radius ($\delta$) to find a state that maintains high accuracy on the source domain but fails catastrophically on the target domain.

Accuracy on source and target domains as the classifier is perturbed by a radius $\delta$. Target accuracy plummets while source accuracy remains stable, revealing poor transferability.

Improving Transferability (The Defense): To mitigate this vulnerability, the authors propose a minimax optimization (adversarial training) algorithm. This method learns a feature embedding that minimizes the worst-case transferability gap among source domains while preserving high accuracy on the source tasks.

Experimental Results

The authors evaluated their methods across several domain generalization benchmarks, including RotatedMNIST, PACS, Office-Home, and the large-scale WILDS-FMoW dataset.

  • Vulnerability of Existing Methods: The adversarial evaluation revealed a glaring weakness in current algorithms like ERM, DANN, and Mixup. For instance, on the PACS dataset using Mixup, slightly perturbing the classifier caused target accuracy to drop by more than 30%, while source accuracy barely changed. This confirms that the underlying features learned by these models are not inherently transferable. Algorithms that naturally learn more robust classifiers, such as Correlation Alignment (CORAL), demonstrated better transferability under this metric.
  • Improved Generalization: The proposed adversarial defense algorithm achieved consistent improvements over state-of-the-art methods across all tested datasets.
  • Tighter Theoretical Bounds: The authors demonstrated that their transfer measure provides a tighter theoretical bound on target error compared to traditional $\mathcal{H}$-divergence, particularly in datasets like PACS where label shift between domains is significant.

Limitations and Future Directions

While effective, the proposed defense algorithm requires a large number of inner maximization steps to compute the adversarial gap, making training significantly slower than standard methods—akin to the overhead seen in adversarial robustness training. It also requires moderate hyperparameter tuning for parameters like the perturbation radius ($\delta$) and the inner optimizer's learning rate.

For future work, the authors suggest exploring "certified robust feature embeddings." Drawing analogies to certified robust classifiers in the adversarial robustness literature, this direction could provide strict mathematical guarantees against distribution shifts in the real world.

Summary updated

Abstract

Out-of-distribution generalization is one of the key challenges when transferring a model from the lab to the real world. Existing efforts mostly focus on building invariant features among source and target domains. Based on invariant features, a high-performing classifier on source domains could hopefully behave equally well on a target domain. In other words, we hope the invariant features to be \emph{transferable}. However, in practice, there are no perfectly transferable features, and some algorithms seem to learn ``more transferable'' features than others. How can we understand and quantify such \emph{transferability}? In this paper, we formally define transferability that one can quantify and compute in domain generalization. We point out the difference and connection with common discrepancy measures between domains, such as total variation and Wasserstein distance. We then prove that our transferability can be estimated with enough samples and give a new upper bound for the target error based on our transferability. Empirically, we evaluate the transferability of the feature embeddings learned by existing algorithms for domain generalization. Surprisingly, we find that many algorithms are not quite learning transferable features, although few could still survive. In light of this, we propose a new algorithm for learning transferable features and test it over various benchmark datasets, including RotatedMNIST, PACS, Office-Home and WILDS-FMoW. Experimental results show that the proposed algorithm achieves consistent improvement over many state-of-the-art algorithms, corroborating our theoretical findings.
BibTeX
@inproceedings{zhang2021quantifying,
  author = {Guojun Zhang and Han Zhao and Yaoliang Yu and Pascal Poupart},
  title = {{Quantifying and Improving Transferability in Domain Generalization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=SQqKl8I6xD8},
}
Figure