Domain Generalization Guided by Large-Scale Pre-Trained Priors

Zongbin Wang, Bin Pan, Shiyu Shen, Tianyang Shi, Zhenwei ShiarXiv 2024 · OpenReview

TL;DR

This paper introduces FT-LP, a novel fine-tuning method for domain generalization (DG) that leverages large-scale pre-trained models as continuous priors throughout the training process. By ensuring the model consistently references its pre-trained knowledge, FT-LP significantly improves generalization to unseen domains, outperforming existing DG techniques.

What Problem Does This Paper Address?

In machine learning, models often assume training and testing data come from the same distribution. However, real-world scenarios frequently involve domain shifts (e.g., changes in weather, style, or texture), which degrade model performance. Domain generalization (DG) aims to train models that can generalize to these unseen target domains using only limited source domains. While current DG methods often initialize models with large pre-trained models, they typically only use this knowledge at the beginning of fine-tuning. The authors argue that this is suboptimal, as pre-trained models already possess domain-invariant capabilities that can be further exploited. The core problem is how to effectively and continuously leverage the rich knowledge embedded in large-scale pre-trained models during the DG fine-tuning process to enhance generalization.

Methods

The paper proposes Fine-Tune with Large-scale pre-trained Priors (FT-LP). The key idea is to incorporate the pre-trained model as a "prior" in the DG optimization objective. This ensures that at each optimization step, the model's learning is guided by its pre-trained knowledge, preventing it from deviating too far from the robust representations it already possesses.

The method has two main components:

  1. Theoretical Framework: The authors derive a new generalization error upper bound for DG problems that explicitly includes the pre-trained prior. They theoretically demonstrate that incorporating this prior can effectively reduce the generalization error. This is framed within the PAC-Bayes theory, which provides a way to bound generalization error.
  2. Implementation Strategy: To make FT-LP practical when only pre-trained weights (not distributions) are available, they propose using an encoder. This encoder simulates the distribution of the pre-trained model, allowing FT-LP to be applied even when only the weights are accessible. They specifically adapt this for Gaussian distributions, using linear encoders to estimate covariance.

Datasets

The experiments were conducted on the Domainbed benchmark, which is a standard evaluation framework for domain generalization. The specific datasets used are:

  • PACS (4 domains, 7 classes)
  • VLCS (4 domains, 5 classes)
  • OfficeHome (4 domains, 65 classes)
  • TerraIncognita (4 domains, 10 classes)
  • DomainNet (6 domains, 345 classes)

Key Insights and Ideas

  • Pre-trained models are more than just initializers: The paper strongly advocates that the knowledge within large pre-trained models is a continuous resource that should be actively leveraged throughout fine-tuning, not just at the start.
  • Continuous prior guidance is key: By treating the pre-trained model as a prior and enforcing its influence at every optimization step, FT-LP helps the model maintain its robust, domain-invariant features.
  • Theoretical grounding for practical gains: The authors provide a theoretical PAC-Bayes bound to justify why this continuous prior guidance improves generalization, connecting theoretical insights to empirical improvements.
  • Bridging the gap between weights and distributions: The proposed encoder mechanism is crucial for applying FT-LP when only pre-trained weights are available, making the method broadly applicable.

Key Results

  • Consistent improvements across DG methods: When combined with FT-LP, 11 different DG algorithms showed improved accuracy across all tested datasets.
  • Significant accuracy gains: The average accuracy improvement across five datasets ranged from 0.2% to 3.3%. For example, on DomainNet, MMD's accuracy increased from 23.4% to 43.1% when combined with FT-LP.
  • Enhanced performance for simpler methods: FT-LP notably boosted the performance of basic ERM (Empirical Risk Minimization), yielding an average increase of 1.2 percentage points.
  • Effectiveness on challenging datasets: The method showed particular strength on TerraIncognita and DomainNet, which are known to be more challenging due to larger domain shifts.
  • Visualization insights: Visualizations (e.g., attention maps) suggest that FT-LP helps models focus on more relevant, invariant features rather than spurious ones, leading to better generalization.

Limitations and Future Work

  • Computational Resources: FT-LP requires additional memory and computational resources due to the covariance encoder, suggesting the need for GPUs (at least 24GB) and potentially multiple GPUs for larger models.
  • Single Prior: The current FT-LP formulation uses a single pre-trained model as a prior. The authors suggest that utilizing multiple pre-trained models simultaneously could further enhance generalization, though this might introduce complexity.
  • Bayesian Optimization: While the theoretical framework is rooted in PAC-Bayes, the implementation uses MAP estimation for computational efficiency. Exploring full Bayesian optimization could offer advantages but comes with higher computational costs and challenges in compatibility with certain DG methods.
  • Encoder Simplification: The covariance encoder is simplified to a linear layer with a diagonal matrix assumption. While this reduces computational burden, it might lead to some loss of information. Future work could explore more complex encoders.
  • Pre-trained Model Selection: The paper suggests that choosing a pre-trained model whose source domain characteristics are closer to the target domain can be more advantageous. Further research could explore more sophisticated methods for selecting optimal pre-trained priors.

Summary updated

Abstract

Domain generalization (DG) aims to train a model from limited source domains, allowing it to generalize to unknown target domains. Typically, DG models only employ large-scale pre-trained models during the initialization of fine-tuning. However, large-scale pre-trained models already possess the ability to resist domain shift. If we reference pre-trained models continuously during fine-tuning to maintain this ability, it could further enhance the generalization ability of the DG model. For this purpose, we introduce a new method called Fine-Tune with Large-scale pre-trained Priors (FT-LP), which incorporates the pre-trained model as a prior into the DG fine-tuning process, ensuring that the model refers to its pre-trained model at each optimization step. FT-LP comprises a theoretical framework and a simple implementation strategy. In theory, we verify the rationality of FT-LP by introducing a generalization error bound with the pre-trained priors for DG. In implementation, we utilize an encoder to simulate the model distribution, enabling the use of FT-LP when only pre-trained weights are available. In summary, we offer a new fine-tuning method for DG algorithms to utilize pre-trained models throughout the fine-tuning process. Through experiments on various datasets and DG models, our proposed method exhibits significant improvements, indicating its effectiveness.
BibTeX
@misc{wang2024domain,
  author = {Zongbin Wang and Bin Pan and Shiyu Shen and Tianyang Shi and Zhenwei Shi},
  title = {{Domain Generalization Guided by Large-Scale Pre-Trained Priors}},
  year = {2024},
  eprint = {2406.05628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.05628},
}
Figure