Is Large-Scale Pretraining the Secret to Good Domain Generalization?

Piotr Teterwak, Kuniaki Saito, Theodoros Tsiligkaridis, Bryan A. Plummer, Kate Saenko

Abstract

Multi-Source Domain Generalization (DG) aims to train models on multiple source domains that perform well on unseen target domains. While recent benchmark scores have improved significantly using web-scale pretrained backbones, it remains unclear if these gains stem from better DG algorithms or simply stronger pre-training. We introduce the Alignment Hypothesis, which posits that DG performance is primarily determined by the initial alignment between image and class label text embeddings in the pretrained model. By splitting evaluation data into "In-pretraining" (IP) and "Out-of-pretraining" (OOP) subsets, we demonstrate that current DG methods excel on IP data but struggle significantly on OOP data, highlighting a critical limitation in their ability to learn new generalizable features.

Introduction: The "Holy Grail" of Robustness

Domain Generalization (DG) is a fundamental challenge in computer vision: how can we train a model on a few "source" domains (e.g., photos and sketches) so that it generalizes to a completely "unseen" target domain (e.g., thermal images)?

In recent years, performance on standard benchmarks like DomainBed has skyrocketed. However, nearly all top-performing methods now utilize massive foundation models like CLIP as backbones. This raises a critical question: are we developing better generalization algorithms, or are we simply measuring the model's ability to recall information from its billion-image pre-training phase? If the "unseen" target domain was effectively seen during pre-training, we are measuring memory, not generalization.

The Alignment Hypothesis

To investigate this, we move beyond visual similarity and propose the Alignment Hypothesis. We define an Alignment Score as the cosine similarity between a pretrained model's image embedding and its corresponding class text embedding (e.g., the vector for a dog image vs. the text "a photo of a dog").

Visualizing the Alignment Hypothesis: DG methods successfully align 'In-Pretraining' data but fail to bridge the gap for 'Out-Of-Pretraining' samples.

Based on this score, we categorize data into two groups:

  1. In-Pretraining (IP): Samples that the model already aligns well with their labels.
  2. Out-of-Pretraining (OOP): Samples where the initial alignment is weak.

Our hypothesis states that final DG performance is high if and only if the initial alignment is high. In other words, the pretrained starting point is the primary driver of success.

Experimental Setup

We evaluated five major DG benchmarks from the DomainBed suite: PACS, OfficeHome, VLCS, TerraIncognita, and DomainNet. To ensure a clean analysis, we filtered out mislabeled data (often found at very low alignment scores) and images containing text that might allow the model to "cheat" via OCR. We introduce DomainBed-OOP, a framework designed to isolate these "hard" samples and provide a more rigorous test of a DG method's learning capacity.

Key Results and Insights

1. Alignment is Destiny

The initial alignment score is a near-perfect predictor of final accuracy. If a model starts with high alignment (IP), it finishes with high accuracy. However, if it starts low (OOP), fine-tuning on source data rarely bridges the gap. On the DomainBed-IP split, top methods like CLIPood achieve ~84.7% accuracy, but this craters to 57.1% on the DomainBed-OOP split.

2. The Generalization Gap

Current DG methods act more like "regularized fine-tuning" than true generalization learners. They are excellent at preserving the features CLIP already possesses but surprisingly poor at synthesizing new robust features from the provided source domains. On OOP data, state-of-the-art methods often fail to consistently outperform simple baselines like Empirical Risk Minimization (ERM).

3. The Oracle Paradox and Zero-Shot Power

On IP data, some DG methods actually outperform an "Oracle" model trained directly on target data, suggesting that pretrained features are sometimes more reliable than the target signal itself. Conversely, in datasets like PACS and VLCS, a raw Zero-Shot CLIP model can outperform fine-tuned DG models on the IP subset, indicating that fine-tuning can sometimes lead to "catastrophic forgetting" of useful pretrained knowledge.

4. Contextual Challenges

Using LLM-based analysis (VisDiff), we found that OOP samples often involve confusing contexts—such as a "car" image dominated by historical architecture or a "bed" in a child-themed room. While CLIP struggles with these nuances, current DG methods fail to learn the necessary features to overcome these contextual shifts.

Conclusion

Our findings suggest that the recent "success" in Domain Generalization is largely an artifact of powerful pre-training. The community must move beyond chasing marginal gains on aggregate benchmarks and focus on the OOP gap. Future research should prioritize methods that can actually learn and generalize beyond the initial alignment of their pretrained backbones, rather than simply anchoring to them.

Summary updated

Abstract

Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. However, it remains unclear if DG finetuning methods are becoming better over time, or if improved benchmark performance is simply an artifact of stronger pre-training. Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot performance, but we find the effect limited in the DG setting. Instead, we posit that having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to introduce the Alignment Hypothesis, which states that the final DG performance will be high if and only if alignment of image and class label text embeddings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it as an analysis tool of existing DG methods evaluated on DomainBed datasets by splitting evaluation data into In-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP. Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment.
BibTeX
@inproceedings{teterwak2025is,
  author = {Piotr Teterwak and Kuniaki Saito and Theodoros Tsiligkaridis and Bryan A. Plummer and Kate Saenko},
  title = {{Is Large-Scale Pretraining the Secret to Good Domain Generalization?}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=wCOJpXm0Me},
}
Figure