Preserving Silent Features for Domain Generalization

Chujie Zhao, Tianren Zhang, Feng ChenarXiv 2024 · DOI · arXiv

Introduction: The Paradox of Self-Supervised Learning

In the field of Domain Generalization (DG), the objective is to train a model on several "source" domains (e.g., photos of dogs) such that it performs robustly on an "unseen" target domain (e.g., sketches of dogs). While standard practice involves fine-tuning an ImageNet-pre-trained model on source domains, a persistent paradox has emerged: self-supervised learning (SSL) models, which intuitively capture more diverse details by learning without labels, often perform worse than supervised models in DG settings.

The authors identify the culprit as feature suppression. During standard fine-tuning, the model's strong label bias forces it to focus exclusively on "dominant features"—the most obvious signals that distinguish classes in the training data (like color). In this process, the model discards subtle but highly generalizable details, termed silent features (like texture or shape), which are essential for cross-domain performance.

Theoretical framework of training vs. test domains highlighting silent (z_s) and dominant (z_d) features.

The STEP Methodology

To address this, the paper introduces STEP (Silent Feature Preservation), a two-stage fine-tuning strategy designed to protect these valuable silent features from being suppressed during adaptation. STEP combines two primary techniques:

  1. LP-FT (Linear Probing then Fine-Tuning): Instead of updating the entire network immediately, the model first freezes the backbone and trains only the classification head (Linear Probing). This allows the model to adapt to new labels without distorting the rich feature representations learned during pre-training. Only after this initial alignment is the entire network unfrozen for full Fine-Tuning.
  2. SWAD (Stochastic Weight Averaging Densely): To prevent the model from converging on "sharp" local minima that overfit the training data, the authors employ SWAD. This technique averages model weights over many iterations during fine-tuning to find "flat" minima, which are empirically linked to better generalization across distribution shifts.

Key Insights: SSL as an Information "Hoarder"

A central contribution of this work is the conceptualization of supervised learning as "greedy" and SSL as a "hoarder." Supervised learning only retains what is necessary to solve the immediate task, leading to feature collapse. Conversely, SSL captures a vast array of features that may seem redundant during training but become vital when the environment changes.

Comparison of feature spaces: Supervised pre-training collapses features, while SSL preserves diverse silent features like shape and texture.

As illustrated above, fine-tuning acts as a double-edged sword. While it helps a model learn a specific task, it aggressively filters out pre-trained features not directly related to the training labels. The authors found that the SSL + LP-FT combination is particularly effective because SSL models still possess these silent features, whereas supervised models have often already lost them during their initial ImageNet training.

Experimental Results

The proposed STEP-S (utilizing a SwAV SSL backbone) was evaluated on five major DG benchmarks: VLCS, PACS, Office-Home, TerraInc, and DomainNet. The results demonstrate state-of-the-art performance:

  • Superior Accuracy: STEP-S outperformed 16 baselines, including standard Empirical Risk Minimization (ERM) and the original SWAD.
  • Significant Gains: On average, it improved accuracy by 3.8 percentage points (pp) over ERM and 1.3 pp over the previous state-of-the-art.
  • Robustness to Style Shifts: The most dramatic improvements were seen on datasets with heavy distribution shifts, such as TerraInc (+5.6 pp) and PACS (+2.0 pp).

Conclusion and Future Directions

STEP provides a simple yet powerful framework for leveraging the rich representations of SSL models for domain generalization. By preventing the suppression of "silent features," the model maintains the diverse information necessary to navigate unseen domains. Future work may involve dynamically balancing the weights of dominant and silent features based on the expected degree of distribution shift, further refining the model's ability to generalize without carrying unnecessary "dead weight" information.

Summary updated

Abstract

Domain generalization (DG) aims to improve the generalization ability of the model trained on several known training domains over unseen test domains. Previous work has shown that self-supervised contrastive pre-training improves the robustness of the model on downstream tasks. However, in this paper, we find that self-supervised models do not exhibit better generalization performance than supervised models pre-trained on the same dataset in the DG setting. We argue that this is owing to the fact that the richer intra-class discriminative features extracted by self-supervised contrastive learning, which we term silent features, are suppressed during supervised fine-tuning. These silent features are likely to contain features that are more generalizable on the test domain. In this work, we model and analyze this feature suppression phenomenon and theoretically prove that preserving silent features can achieve lower expected test domain risk under certain conditions. In light of this, we propose a simple yet effective method termed STEP (Silent Feature Preservation) to improve the generalization performance of the self-supervised contrastive learning pre-trained model by alleviating the suppression of silent features during the supervised fine-tuning process. Experimental results show that STEP exhibits state-of-the-art performance on standard DG benchmarks with significant distribution shifts.
BibTeX
@misc{zhao2024preserving,
  author = {Chujie Zhao and Tianren Zhang and Feng Chen},
  title = {{Preserving Silent Features for Domain Generalization}},
  year = {2024},
  eprint = {2401.03170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.03170},
}
Figure