Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models

Byounggyu Lew, Donghyun Son, Buru Chang

Abstract

Domain generalization aims to build generalized models that perform well on unseen domains when only source domains are available for model optimization. Recent studies have shown that large-scale pre-trained models can enhance domain generalization by leveraging their generalization power. However, these pre-trained models lack target task-specific knowledge yet due to discrepancies between the pre-training objectives and the target task. Although the task-specific knowledge could be learned from source domains by fine-tuning, this hurts the generalization power of pre-trained models due to gradient bias toward the source domains. To alleviate this problem, we propose a new domain generalization method that estimates unobservable gradients that reduce potential risks in unseen domains using a large-scale pre-trained model. These estimated unobservable gradients allow the pre-trained model to learn task-specific knowledge further while preserving its generalization ability by relieving the gradient bias. Our experimental results show that our method outperforms baseline methods on DOMAINBED, a standard benchmark in domain generalization.

Introduction

In the real world, machine learning models often encounter data from distributions different from their training sets. Domain Generalization (DG) seeks to create models capable of performing well on these "unseen" domains. While large-scale pre-trained models (PTMs) like CLIP or ResNet-50 possess significant inherent generalization capabilities, they often lack the specific knowledge required for a particular target task.

The standard approach to address this is fine-tuning on available source domains. However, fine-tuning often introduces gradient bias. The optimization process pushes model parameters solely toward minimizing risk on source data, which can inadvertently destroy the generalized representations originally present in the PTM. This phenomenon is characterized by "gradient conflicts," where the direction required to improve on source data negatively impacts performance on unseen data.

The GESTUR Framework

To mitigate gradient bias, we introduce GESTUR (Gradient Estimation for Unseen Domain Risk Minimization). The core idea is to estimate the "unobservable gradients" ($g_u$) that would ideally guide the model toward better performance on unseen domains.

Conceptual overview of GESTUR: estimating unobserved gradients (g_u) to mitigate biased gradients (g) from source domains.

GESTUR utilizes a dual-model architecture:

  1. Task Expert (TE): Learns task-specific knowledge directly from source domains.
  2. Generalization Expert (GE): Updated via an Exponential Moving Average (EMA) of the Task Expert’s parameters. By changing slowly, the GE preserves more of the original PTM’s generalization ability.

We hypothesize that the vector pointing from the Task Expert toward the Generalization Expert serves as a "loose approximation" of the unobservable gradient. During training, the Task Expert follows a combined gradient that incorporates this estimated signal, effectively steering the optimization away from training-data bias.

Experimental Evaluation

We evaluated GESTUR on the DomainBed benchmark, covering five diverse datasets: PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet. We tested the method across three distinct pre-trained backbones: ResNet-50 (ImageNet), ViT-B/16 (CLIP), and RegNetY-16GF (SWAG).

Performance comparison of GESTUR against baseline methods on the DomainBed benchmark across various backbones.

As shown in the results, GESTUR consistently outperforms standard Empirical Risk Minimization (ERM) and other state-of-the-art methods like SWAD and MIRO. Notably, the performance gap between GESTUR and baselines increases as the pre-trained model quality improves (e.g., moving from ResNet to CLIP), suggesting that GESTUR is particularly effective at unlocking the potential of massive foundation models.

Key Insights

  • The "Loose Oracle" Assumption: By treating the GE as a version of the model that stays closer to the PTM's original "oracle" state, we can use it as a compass to guide the more aggressive learner (the TE).
  • Reducing Gradient Conflicts: Quantitative analysis shows that GESTUR significantly reduces instances where the source gradient conflicts with the direction needed for generalization.
  • Task-Specific Learning: Ablation studies confirm that the GE does not remain static; through the EMA process, it learns task-specific knowledge while remaining a better approximation of a generalized model than a frozen PTM.

Conclusion

GESTUR provides a robust mechanism for fine-tuning large-scale pre-trained models without sacrificing their inherent generalization power. By estimating unobservable gradients, it effectively navigates the trade-off between task-specific expertise and domain-agnostic robustness, achieving new state-of-the-art results in domain generalization.

Summary updated

Abstract

Domain generalization aims to build generalized models that perform well on unseen domains when only source domains are available for model optimization. Recent studies have shown that large-scale pre-trained models can enhance domain generalization by leveraging their generalization power. However, these pre-trained models lack target task-specific knowledge yet due to discrepancies between the pre-training objectives and the target task. Although the task-specific knowledge could be learned from source domains by fine-tuning, this hurts the generalization power of pre-trained models due to gradient bias toward the source domains. To alleviate this problem, we propose a new domain generalization method that estimates unobservable gradients that reduce potential risks in unseen domains using a large-scale pre-trained model. These estimated unobservable gradients allow the pre-trained model to learn task-specific knowledge further while preserving its generalization ability by relieving the gradient bias. Our experimental results show that our method outperforms baseline methods on DOMAINBED, a standard benchmark in domain generalization. We also provide extensive analyses to demonstrate that the pre-trained model can learn task-specific knowledge without sacrificing its generalization power.
BibTeX
@misc{lew2023gradient,
  author = {Byounggyu Lew and Donghyun Son and Buru Chang},
  title = {{Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models}},
  year = {2023},
  eprint = {2302.01497},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.01497},
}
Figure