Abstract
Machine learning models can suffer from performance degradation when applied to new tasks due to distribution shifts. Feature representation learning offers a robust solution to this issue. However, a fundamental challenge remains in devising the optimal strategy for feature selection. Existing literature is somewhat paradoxical: some advocate for learning invariant features from source domains, while others favor more diverse features. For better understanding, we propose a statistical framework that evaluates the utilities of the features based on the variance of their correlation to $y$ across different domains. Under our framework, we design and analyze a learning procedure consisting of learning content features (comprising both invariant and approximately shared features) from source tasks and fine-tuning them on the target task. Our theoretical analysis highlights the significance of learning approximately shared features—beyond strictly invariant ones—when distribution shifts occur. Our analysis also yields an improved population risk on target tasks compared to previous results. Inspired by our theory, we introduce ProjectionNet, a practical method to distinguish content features from environmental features via \textit{explicit feature space control}, further consolidating our theoretical findings.
Introduction
Machine learning models are notoriously brittle when deployment data (the target domain) differs from training data (source domains), a phenomenon known as distribution shift. To address this, researchers typically aim to learn robust feature representations. However, a conflict exists between two schools of thought: the "Invariant Camp," which prioritizes features perfectly consistent across domains, and the "Diverse Camp," which argues that a wider variety of features—even those that vary slightly—enhances robustness. This paper bridges this gap by identifying "approximately shared features" as a critical component for robust learning.
Methods
The authors categorize features based on the variance of their correlation with the target label ($y$) across environments ($e$).
Feature Taxonomy
- Content Features: Useful for prediction, including Invariant Features (stable across domains) and Approximately Shared Features (correlated with both label and environment, providing useful signal despite variance).
- Environmental (Spurious) Features: Correlated with the environment but uncorrelated with the label on average; these act as distractors.
ProjectionNet Architecture
ProjectionNet implements this theory by using a shared backbone and three orthogonal projection matrices to disentangle representations into target-specific ($\Phi_y$), approximately shared ($\Phi_{y,e}$), and environment-specific ($\Phi_e$) components.
Results
The approach was evaluated on standard benchmarks including OfficeHome, PACS, VLCS, and TerrainCognita.
Performance and Efficiency
ProjectionNet achieves performance comparable to or exceeding state-of-the-art methods like DIWA, while requiring 20 times less training time. In domain generalization tasks, it consistently outperformed standard Empirical Risk Minimization (ERM) and domain-adversarial methods (DANN).
Interpretability
GradCAM++ visualizations demonstrate that ProjectionNet focuses on semantically meaningful parts of an image (e.g., the entire object) rather than local textures or background elements favored by other methods.
Conclusion
This work demonstrates that strictly invariant features are often insufficient for fast adaptation. Approximately shared features provide the richness necessary for a model to adapt to new domains with minimal samples. By explicitly controlling the feature space, ProjectionNet offers a robust and efficient solution for domain adaptation. Future work includes scaling these disentanglement techniques to large-scale foundation models.
Summary updated