Bridging Domains with Approximately Shared Features

Ziliang Samuel Zhong, Xiang Pan, Qi Lei

Abstract

Machine learning models often fail to generalize across domains due to the "invariance paradox": the choice between discarding useful information to achieve strict invariance or risking spurious correlations for diversity. This paper introduces a middle ground by identifying approximately shared features—content-related features with low variance across domains. We propose ProjectionNet, which disentangles these features using specialized projection heads. Our approach achieves state-of-the-art performance on the DomainBed benchmark while being 20x faster to train than leading ensemble methods like DiWA.

Introduction: The Invariance Paradox

A persistent challenge in machine learning is the failure of models to generalize across different domains, such as transitioning from photographic images to artistic sketches. This problem, addressed by Multi-Source Domain Adaptation (MSDA), has historically been viewed through two conflicting lenses:

  1. The Invariant Perspective: Advocates for learning only those features that remain perfectly identical across all source domains (e.g., the geometric arrangement of clock hands). This often results in an "information deficit," where useful but non-invariant data is ignored.
  2. The Diverse Perspective: Seeks to capture as many features as possible to ensure coverage. However, it risks latching onto "spurious" or environmental features (e.g., background textures) that do not generalize.

This paper proposes a middle ground: identifying and utilizing features that are "content-rich" but not necessarily perfectly invariant.

A New Taxonomy of Features

The authors provide a theoretical framework that categorizes features based on their correlation variance with the label $y$ across environments.

Visualizing the feature taxonomy: Invariant features (clock hands), approximately shared features (clock face), and spurious features (background) across different domains, alongside GradCAM++ visualizations of various models.

In this framework, features are split into three categories:

  • Strictly Invariant Features: Zero variance in their relationship to the label across domains.
  • Approximately Shared Features: Mostly stable but vary slightly. These are crucial for rapid adaptation to new domains.
  • Environmental/Spurious Features: High variance and zero mean correlation, representing domain-specific noise.

By accounting for approximately shared features, the authors prove that the "irreducible error floor" often found in representation learning theory can be eliminated as more target samples become available.

ProjectionNet: Disentangling the Feature Space

To operationalize this theory, the authors developed ProjectionNet (PN). The architecture employs a shared feature encoder followed by three distinct projection heads designed to isolate specific feature types.

The ProjectionNet architecture, illustrating the disentanglement of features into invariant ($\phi_y$), approximately shared ($\phi_s$), and environment-specific ($\phi_e$) components.

The model is trained to predict both the class label $y$ (using the $\phi_y$ and $\phi_s$ heads) and the environment label $e$ (using the $\phi_s$ and $\phi_e$ heads). This multi-task objective forces the network to disentangle "content" from "environment." During fine-tuning on a target task, the model prioritizes the learned content features, leading to more robust generalization.

Experimental Evaluation

The proposed method was evaluated on the DomainBed benchmark, covering diverse datasets including OfficeHome, PACS, TerrainCognita, and VLCS. These datasets present various challenges, from stylistic shifts to geographic shifts in camera trap data.

Performance and Efficiency

ProjectionNet demonstrated significant advantages over existing methods:

  • State-of-the-Art Accuracy: On the PACS dataset, the PN-Y and PN-YS variants consistently outperformed standard Empirical Risk Minimization (ERM) and strict invariance methods like DANN, particularly in few-shot scenarios.
  • Training Speed: Remarkably, ProjectionNet achieved results comparable to or better than DiWA (a leading model ensemble method) while requiring 20x less training time. This makes it a highly practical solution for large-scale applications.

Interpretability

Qualitative analysis using GradCAM++ visualizations (as seen in the feature taxonomy figure) revealed that ProjectionNet focuses on semantically relevant parts of an image—such as the face of a clock—whereas standard ERM models often focus on local textures or background elements that are specific to the training domain.

Conclusion

By bridging the gap between strict invariance and total diversity, the "approximately shared" feature framework provides a more nuanced and effective approach to domain adaptation. ProjectionNet offers a computationally efficient way to implement this theory, providing a robust path forward for building models that can truly generalize across the diverse domains of the real world.

Summary updated

BibTeX
@inproceedings{zhong2025bridging,
  author = {Ziliang Samuel Zhong and Xiang Pan and Qi Lei},
  title = {{Bridging Domains with Approximately Shared Features}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=25oDoZG4a5},
}
Figure