Introduction
Deep neural networks often exhibit a "simplicity bias," where they prioritize learning the easiest patterns—often referred to as shortcuts—to minimize training loss. While effective within the training domain, these shortcuts frequently fail to generalize to new environments, leading to significant performance degradation under domain shift. This paper introduces L2GP (Learning Less Generalizable Patterns), a training strategy designed to counteract this bias by intentionally seeking out more complex, diverse features that facilitate better Test-Time Adaptation (TTA).
The L2GP Framework
The core objective of L2GP is to force the model to learn a "backup" of diverse features that might be ignored by standard training. The architecture consists of a shared feature extractor $f$ and two distinct classifier heads, $c_1$ and $c_2$.
Asymmetric Training
The training process is asymmetric. While $c_1$ is trained using standard Cross-Entropy loss to capture primary generalizable features, $c_2$ acts as an "explorer." It employs a novel Shortcut Avoidance Loss that penalizes the model if its predictions on a different batch change significantly after a small weight perturbation in the direction of the current batch's gradient.
The "Aha" moment of this approach lies in this batch-wise comparison. By penalizing the model when a perturbation based on one batch affects predictions on another, the system is forced to identify patterns that are predictive for specific samples but do not necessarily dominate the overall loss. This mechanism forces the feature extractor to become a much richer representation of the data, capturing "harder," less obvious features that it would normally skip.
Experimental Setup
The authors evaluate L2GP on two major benchmarks for Single-Source Domain Generalization (SSDG):
- PACS: A standard benchmark containing Photos, Art, Cartoons, and Sketches across 7 classes.
- Office-Home: A more challenging dataset with 65 classes across four domains (Art, Clipart, Product, and Real-World). This dataset is rarely used in single-source studies, making these results particularly noteworthy.
At test-time, the second head is discarded. The model utilizes the shared feature extractor $f$ and the primary head $c_1$, combined with Test-Time Batch Normalization (TTBN), which updates internal statistics based on the new data encountered.
Results and Analysis
L2GP demonstrates superior performance across both benchmarks, particularly when paired with TTBN. The results highlight a significant synergy between L2GP's training objective and test-time adaptation.
On the challenging Office-Home dataset, L2GP achieves a test accuracy of 54.5% with TTBN, outperforming the standard Empirical Risk Minimization (ERM) baseline of 52.6% and other specialized methods like EFDM. On PACS, it reaches 71.3% accuracy, matching the state-of-the-art while showing higher stability across different training runs.
Interestingly, L2GP also improved accuracy on the validation set of the original training domain. This suggests that learning "hidden" patterns helps even when the domain doesn't shift, as it provides a more comprehensive understanding of the data. However, the authors note that the method's gains are most pronounced when paired with TTBN, and performance can be sensitive to hyperparameters like batch size and perturbation step size.
Conclusion
L2GP addresses the limitations of shortcut learning by explicitly encouraging the discovery of "less generalizable" but potentially more robust features. By combining this enriched representation with test-time adaptation, the method achieves state-of-the-art results in single-source domain generalization. This work highlights that intentional "bad" learning—forcing a model to find patterns it would usually ignore—is a powerful tool for building models that can adapt to the unpredictable shifts of the real world.
Summary updated