PGrad: Learning Principal Gradients For Domain Generalization

Zhe Wang, Jake Grigsby, Yanjun Qi

Deep neural networks are notoriously "lazy": they often exploit easy, domain-specific correlations (like background textures) rather than learning the actual underlying features (like object shapes). For example, a model might learn to recognize a "camel" only when it sees a desert background, failing completely if it encounters a camel in a green pasture.

This is the core challenge of Domain Generalization (DG). Standard Empirical Risk Minimization (ERM) averages gradients across domains, but this often preserves domain-specific noise that hinders performance on out-of-distribution (OOD) test data.

To address this, researchers introduce PGrad, a novel training strategy that improves a model's ability to generalize to unseen environments. By extracting a "principal gradient" from an optimization trajectory across multiple training domains, PGrad filters out domain-specific noise and follows the most robust directions of parameter change.

How PGrad Works

PGrad moves away from standard parallel gradient averaging in favor of a trajectory-based approach. The method can be broken down into a few key steps:

1. Trajectory Sampling (Roll-out) Starting from the current model parameters, the algorithm performs a "roll-out" by sequentially optimizing the model on each training domain one after another. This creates a path (trajectory) in the high-dimensional parameter space.

Sequential trajectory sampling across domains

2. Principal Coordinate Construction PGrad treats the points in this trajectory as data and uses Singular Value Decomposition (SVD) to find the "principal directions" (eigenvectors) of parameter variation. To avoid performing SVD on millions of parameters, the authors use a mathematical bijection trick, allowing them to compute eigenvectors using a small matrix related to the number of domains rather than the number of parameters.

3. Noise Suppression and the "Principal Gradient" A new gradient direction is constructed as a weighted sum of these principal directions. Crucially, PGrad employs a noise suppression step by only aggregating the top-$k$ eigenvectors. This effectively filters out the low-variance "tail" directions that correspond to domain-specific noise.

Constructing the principal gradient with noise suppression

4. Calibration Finally, the resulting vector is calibrated in both direction (to ensure it aligns with the overall optimization goal) and length (to allow for adaptive step sizes).

Key Insights

  • Sequential vs. Parallel: A major insight of this work is that sequential training (one domain after another) reinforces "clean" shared directions. In contrast, parallel training (calculating all gradients from the same starting point) can actually suppress these robust signals during the centering process.
  • Spectral Connection: The authors theoretically prove that PGrad is essentially performing a spectral analysis of the Fisher Information Matrix (FIM), which approximates the Hessian (the curvature of the loss landscape). This connects DG to the idea that neural networks optimize most efficiently in a low-dimensional "top subspace."
  • Tail Eigenvectors are Toxic: Experiments show that the "bottom" eigenvectors—those with the smallest eigenvalues—contain signals that help accuracy on specific training domains but actively hurt generalization to new ones. Filtering them out is key to robustness.
  • Model Agnostic: Because PGrad operates on the gradient/parameter level, it can be wrapped around almost any architecture (ResNet, DenseNet, etc.) and combined with other DG techniques like MixUp.

Performance and Results

The authors evaluated PGrad on two major suites of benchmarks: DomainBed (including PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet) and WILDS (focusing on real-world distributional shifts like PovertyMap and FMoW).

  • State-of-the-Art Accuracy: PGrad-BMix (a variant combining PGrad with batch splitting and MixUp) achieved the highest average accuracy (65.7%) across seven datasets on DomainBed, outperforming ERM and ten other established DG algorithms. On the WILDS datasets, PGrad-B set new state-of-the-art marks for complex, real-world satellite imagery tasks.
  • Optimization Stability: Visualizations of the loss curves show that PGrad leads to much smoother optimization compared to ERM. While ERM's loss on one domain often spikes while it improves on another, PGrad allows all domains to improve in unison.

Limitations and Future Work

While highly effective, PGrad introduces a new hyperparameter $k$ (the number of principal components to keep). The optimal $k$ might vary depending on the diversity of the training domains. Additionally, the "roll-out" trajectory requires sequential forward/backward passes for each domain per update, which can increase training time compared to simple ERM. Future work may explore more sophisticated ways to sample trajectories or automatically adapt the noise suppression threshold during training.

Summary updated

Abstract

Machine learning models fail to perform when facing out-of-distribution (OOD) domains, a challenging task known as domain generalization (DG). In this work, we develop a novel DG training strategy, we call PGrad, to learn a robust gradient direction, improving models' generalization ability on unseen domains. The proposed gradient aggregates the principal directions of a sampled roll-out optimization trajectory that measures the training dynamics across all training domains. PGrad gradient design forces the DG training to ignore domain-dependent noise signals and updates all training domains with a robust direction covering main components of parameter dynamics. We further improve PGrad via bijection-based computational refinement and directional plus length-based calibrations. Our theoretical proof connects PGrad to the spectral analysis of Hessian in training neural networks. Experiments on DomainBed and WILDS benchmarks demonstrate that our approach effectively enables robust DG optimization and leads to smoothly decreased loss curves. Empirically, PGrad achieves competitive results across seven datasets, demonstrating its efficacy across both synthetic and real-world distributional shifts.
BibTeX
@inproceedings{wang2023pgrad,
  author = {Zhe Wang and Jake Grigsby and Yanjun Qi},
  title = {{PGrad: Learning Principal Gradients For Domain Generalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=CgCmwcfgEdH},
}
Figure