Abstract
Multi-Source Domain Generalization (DG) aims to train models on multiple source domains that perform well on unseen target domains. While recent benchmark scores have improved significantly using web-scale pretrained backbones, it remains unclear if these gains stem from better DG algorithms or simply stronger pre-training. We introduce the Alignment Hypothesis, which posits that DG performance is primarily determined by the initial alignment between image and class label text embeddings in the pretrained model. By splitting evaluation data into "In-pretraining" (IP) and "Out-of-pretraining" (OOP) subsets, we demonstrate that current DG methods excel on IP data but struggle significantly on OOP data, highlighting a critical limitation in their ability to learn new generalizable features.
Introduction: The "Holy Grail" of Robustness
Domain Generalization (DG) is a fundamental challenge in computer vision: how can we train a model on a few "source" domains (e.g., photos and sketches) so that it generalizes to a completely "unseen" target domain (e.g., thermal images)?
In recent years, performance on standard benchmarks like DomainBed has skyrocketed. However, nearly all top-performing methods now utilize massive foundation models like CLIP as backbones. This raises a critical question: are we developing better generalization algorithms, or are we simply measuring the model's ability to recall information from its billion-image pre-training phase? If the "unseen" target domain was effectively seen during pre-training, we are measuring memory, not generalization.
The Alignment Hypothesis
To investigate this, we move beyond visual similarity and propose the Alignment Hypothesis. We define an Alignment Score as the cosine similarity between a pretrained model's image embedding and its corresponding class text embedding (e.g., the vector for a dog image vs. the text "a photo of a dog").
Based on this score, we categorize data into two groups:
- In-Pretraining (IP): Samples that the model already aligns well with their labels.
- Out-of-Pretraining (OOP): Samples where the initial alignment is weak.
Our hypothesis states that final DG performance is high if and only if the initial alignment is high. In other words, the pretrained starting point is the primary driver of success.
Experimental Setup
We evaluated five major DG benchmarks from the DomainBed suite: PACS, OfficeHome, VLCS, TerraIncognita, and DomainNet. To ensure a clean analysis, we filtered out mislabeled data (often found at very low alignment scores) and images containing text that might allow the model to "cheat" via OCR. We introduce DomainBed-OOP, a framework designed to isolate these "hard" samples and provide a more rigorous test of a DG method's learning capacity.
Key Results and Insights
1. Alignment is Destiny
The initial alignment score is a near-perfect predictor of final accuracy. If a model starts with high alignment (IP), it finishes with high accuracy. However, if it starts low (OOP), fine-tuning on source data rarely bridges the gap. On the DomainBed-IP split, top methods like CLIPood achieve ~84.7% accuracy, but this craters to 57.1% on the DomainBed-OOP split.
2. The Generalization Gap
Current DG methods act more like "regularized fine-tuning" than true generalization learners. They are excellent at preserving the features CLIP already possesses but surprisingly poor at synthesizing new robust features from the provided source domains. On OOP data, state-of-the-art methods often fail to consistently outperform simple baselines like Empirical Risk Minimization (ERM).
3. The Oracle Paradox and Zero-Shot Power
On IP data, some DG methods actually outperform an "Oracle" model trained directly on target data, suggesting that pretrained features are sometimes more reliable than the target signal itself. Conversely, in datasets like PACS and VLCS, a raw Zero-Shot CLIP model can outperform fine-tuned DG models on the IP subset, indicating that fine-tuning can sometimes lead to "catastrophic forgetting" of useful pretrained knowledge.
4. Contextual Challenges
Using LLM-based analysis (VisDiff), we found that OOP samples often involve confusing contexts—such as a "car" image dominated by historical architecture or a "bed" in a child-themed room. While CLIP struggles with these nuances, current DG methods fail to learn the necessary features to overcome these contextual shifts.
Conclusion
Our findings suggest that the recent "success" in Domain Generalization is largely an artifact of powerful pre-training. The community must move beyond chasing marginal gains on aggregate benchmarks and focus on the OOP gap. Future research should prioritize methods that can actually learn and generalize beyond the initial alignment of their pretrained backbones, rather than simply anchoring to them.
Summary updated