Introduction: The Paradox of Self-Supervised Learning
In the field of Domain Generalization (DG), the objective is to train a model on several "source" domains (e.g., photos of dogs) such that it performs robustly on an "unseen" target domain (e.g., sketches of dogs). While standard practice involves fine-tuning an ImageNet-pre-trained model on source domains, a persistent paradox has emerged: self-supervised learning (SSL) models, which intuitively capture more diverse details by learning without labels, often perform worse than supervised models in DG settings.
The authors identify the culprit as feature suppression. During standard fine-tuning, the model's strong label bias forces it to focus exclusively on "dominant features"—the most obvious signals that distinguish classes in the training data (like color). In this process, the model discards subtle but highly generalizable details, termed silent features (like texture or shape), which are essential for cross-domain performance.
The STEP Methodology
To address this, the paper introduces STEP (Silent Feature Preservation), a two-stage fine-tuning strategy designed to protect these valuable silent features from being suppressed during adaptation. STEP combines two primary techniques:
- LP-FT (Linear Probing then Fine-Tuning): Instead of updating the entire network immediately, the model first freezes the backbone and trains only the classification head (Linear Probing). This allows the model to adapt to new labels without distorting the rich feature representations learned during pre-training. Only after this initial alignment is the entire network unfrozen for full Fine-Tuning.
- SWAD (Stochastic Weight Averaging Densely): To prevent the model from converging on "sharp" local minima that overfit the training data, the authors employ SWAD. This technique averages model weights over many iterations during fine-tuning to find "flat" minima, which are empirically linked to better generalization across distribution shifts.
Key Insights: SSL as an Information "Hoarder"
A central contribution of this work is the conceptualization of supervised learning as "greedy" and SSL as a "hoarder." Supervised learning only retains what is necessary to solve the immediate task, leading to feature collapse. Conversely, SSL captures a vast array of features that may seem redundant during training but become vital when the environment changes.
As illustrated above, fine-tuning acts as a double-edged sword. While it helps a model learn a specific task, it aggressively filters out pre-trained features not directly related to the training labels. The authors found that the SSL + LP-FT combination is particularly effective because SSL models still possess these silent features, whereas supervised models have often already lost them during their initial ImageNet training.
Experimental Results
The proposed STEP-S (utilizing a SwAV SSL backbone) was evaluated on five major DG benchmarks: VLCS, PACS, Office-Home, TerraInc, and DomainNet. The results demonstrate state-of-the-art performance:
- Superior Accuracy: STEP-S outperformed 16 baselines, including standard Empirical Risk Minimization (ERM) and the original SWAD.
- Significant Gains: On average, it improved accuracy by 3.8 percentage points (pp) over ERM and 1.3 pp over the previous state-of-the-art.
- Robustness to Style Shifts: The most dramatic improvements were seen on datasets with heavy distribution shifts, such as TerraInc (+5.6 pp) and PACS (+2.0 pp).
Conclusion and Future Directions
STEP provides a simple yet powerful framework for leveraging the rich representations of SSL models for domain generalization. By preventing the suppression of "silent features," the model maintains the diverse information necessary to navigate unseen domains. Future work may involve dynamically balancing the weights of dominant and silent features based on the expected degree of distribution shift, further refining the model's ability to generalize without carrying unnecessary "dead weight" information.
Summary updated