Introduction
In deep learning, the geometry of the loss landscape is a critical determinant of a model's ability to generalize. "Flat minima"—broad, stable regions in the weight space—are associated with better performance on unseen data compared to "sharp minima," where small perturbations in weights lead to significant increases in error. While optimization techniques designed to find these flat regions have revolutionized computer vision, their application to Graph Neural Networks (GNNs) has remained under-explored.
GNNs present unique challenges due to the non-i.i.d. (independent and identically distributed) nature of graph data and their notorious sensitivity to hyperparameters and data splits. This research systematically explores how flat-minima optimization can stabilize GNNs and identifies why standard benchmarks often fail to capture the benefits of these methods.
Methodology: Seeking Flatness in Graph Landscapes
The authors evaluate a comprehensive suite of flat-minima methods, categorized into two primary strategies:
1. Sharpness-Aware Minimization
These methods modify the training step to explicitly seek out flatter regions of the loss landscape.
- SAM (Sharpness-Aware Minimization): Minimizes the loss at the "worst-case" point in a small neighborhood around the current weights.
- ASAM (Adaptive SAM): Scales the neighborhood based on parameter magnitude to account for weight scale invariance.
- PGN (Penalizing Gradient Norm): Directly penalizes the steepness of the gradient to encourage smoothness.
- SAF (Sharpness-Aware Training for Free): An efficient variant that avoids extra gradient calculations by utilizing trajectory loss.
2. Weight Averaging
These techniques average the weights of the model at different points during the training trajectory.
- SWA (Stochastic Weight Averaging): Averages weights at the end of training to find a more central, flatter point in the optima.
- EWA (Exponential Weight Averaging): Uses a running average that favors more recent weights, proving more robust to early stopping and varying training durations.
The Role of Data Splits
A central contribution of this research is the revelation that standard GNN benchmarking practices—specifically the use of fixed "Planetoid" data splits—can mask the effectiveness of optimization improvements. On fixed splits, models often overfit to the specific, static configuration of training and test nodes.
By employing randomized splits (such as "ra-pl" and "622"), the authors demonstrate that the generalization benefits of flat-minima methods become significantly more apparent. On these randomized splits, the methods improved GNN performance by over 2 percentage points (e.g., GAT+EWA on the CiteSeer dataset). This suggests that the "split" itself is often the bottleneck in measuring true algorithmic progress in the GNN field.
Key Insights and Results
- Architecture-Specific Preferences: There is no "one-size-fits-all" optimizer for GNNs. GCNs performed best with GASAM (a combination of GSAM and ASAM), GATs favored PGN, and Graph-MLP excelled with SAF.
- The Power of EWA: The authors strongly recommend Exponential Weight Averaging (EWA) for practitioners. Unlike SWA, which requires precise timing for the start of averaging, EWA is robust across training durations ranging from 5 to 2,000 epochs.
- Stacking Gains: Combining methods (e.g., EWA + GASAM) often leads to higher performance than using either method in isolation, suggesting that weight averaging and sharpness-aware training provide complementary benefits to the optimization process.
- Inductive vs. Transductive Success: The benefits were consistent across both transductive settings (learning on a fixed graph) and inductive settings (generalizing to unseen graphs, such as the PPI dataset).
Conclusion and Future Work
This work underscores the importance of robust evaluation protocols in GNN research. By moving beyond fixed data splits and adopting flat-minima optimization techniques, practitioners can achieve more stable and higher-performing graph models. While the computational cost of tuning hyperparameters like the neighborhood size ($\rho$) remains a challenge for large-scale graphs like OGB arXiv, the performance gains justify the investment. Future research should investigate whether these optimization benefits extend beyond node classification to tasks like edge prediction and whole-graph classification.
Summary updated