The Split Matters: Flat Minima Methods for Improving the Performance of GNNs

Nicolas Lell, Ansgar Scherp

Introduction

In deep learning, the geometry of the loss landscape is a critical determinant of a model's ability to generalize. "Flat minima"—broad, stable regions in the weight space—are associated with better performance on unseen data compared to "sharp minima," where small perturbations in weights lead to significant increases in error. While optimization techniques designed to find these flat regions have revolutionized computer vision, their application to Graph Neural Networks (GNNs) has remained under-explored.

GNNs present unique challenges due to the non-i.i.d. (independent and identically distributed) nature of graph data and their notorious sensitivity to hyperparameters and data splits. This research systematically explores how flat-minima optimization can stabilize GNNs and identifies why standard benchmarks often fail to capture the benefits of these methods.

Methodology: Seeking Flatness in Graph Landscapes

The authors evaluate a comprehensive suite of flat-minima methods, categorized into two primary strategies:

1. Sharpness-Aware Minimization

These methods modify the training step to explicitly seek out flatter regions of the loss landscape.

  • SAM (Sharpness-Aware Minimization): Minimizes the loss at the "worst-case" point in a small neighborhood around the current weights.
  • ASAM (Adaptive SAM): Scales the neighborhood based on parameter magnitude to account for weight scale invariance.
  • PGN (Penalizing Gradient Norm): Directly penalizes the steepness of the gradient to encourage smoothness.
  • SAF (Sharpness-Aware Training for Free): An efficient variant that avoids extra gradient calculations by utilizing trajectory loss.

2. Weight Averaging

These techniques average the weights of the model at different points during the training trajectory.

  • SWA (Stochastic Weight Averaging): Averages weights at the end of training to find a more central, flatter point in the optima.
  • EWA (Exponential Weight Averaging): Uses a running average that favors more recent weights, proving more robust to early stopping and varying training durations.

The Role of Data Splits

A central contribution of this research is the revelation that standard GNN benchmarking practices—specifically the use of fixed "Planetoid" data splits—can mask the effectiveness of optimization improvements. On fixed splits, models often overfit to the specific, static configuration of training and test nodes.

By employing randomized splits (such as "ra-pl" and "622"), the authors demonstrate that the generalization benefits of flat-minima methods become significantly more apparent. On these randomized splits, the methods improved GNN performance by over 2 percentage points (e.g., GAT+EWA on the CiteSeer dataset). This suggests that the "split" itself is often the bottleneck in measuring true algorithmic progress in the GNN field.

Key Insights and Results

  • Architecture-Specific Preferences: There is no "one-size-fits-all" optimizer for GNNs. GCNs performed best with GASAM (a combination of GSAM and ASAM), GATs favored PGN, and Graph-MLP excelled with SAF.
  • The Power of EWA: The authors strongly recommend Exponential Weight Averaging (EWA) for practitioners. Unlike SWA, which requires precise timing for the start of averaging, EWA is robust across training durations ranging from 5 to 2,000 epochs.
  • Stacking Gains: Combining methods (e.g., EWA + GASAM) often leads to higher performance than using either method in isolation, suggesting that weight averaging and sharpness-aware training provide complementary benefits to the optimization process.
  • Inductive vs. Transductive Success: The benefits were consistent across both transductive settings (learning on a fixed graph) and inductive settings (generalizing to unseen graphs, such as the PPI dataset).

Conclusion and Future Work

This work underscores the importance of robust evaluation protocols in GNN research. By moving beyond fixed data splits and adopting flat-minima optimization techniques, practitioners can achieve more stable and higher-performing graph models. While the computational cost of tuning hyperparameters like the neighborhood size ($\rho$) remains a challenge for large-scale graphs like OGB arXiv, the performance gains justify the investment. Future research should investigate whether these optimization benefits extend beyond node classification to tasks like edge prediction and whole-graph classification.

Summary updated

Abstract

When training a Neural Network, it is optimized using the available training data with the hope that it generalizes well to new or unseen testing data. At the same absolute value, a flat minimum in the loss landscape is presumed to generalize better than a sharp minimum. Methods for determining flat minima have been mostly researched for independent and identically distributed (i.i.d.) data such as images. Graphs are inherently non-i.i.d. since the vertices are edge-connected. We investigate flat minima methods and combinations of those methods for training graph neural networks (GNNs). We use GCN and GAT as well as extend Graph-MLP to work with more layers and larger graphs. We conduct experiments on small and large citation, co-purchase, and protein datasets with different train-test splits in both the transductive and inductive training procedure. Results show that flat minima methods can improve the performance of GNN models by over 2 points, if the train-test split is randomized. Following Shchur et al., randomized splits are essential for a fair evaluation of GNNs, as other (fixed) splits like “Planetoid” are biased. Overall, we provide important insights for improving and fairly evaluating flat minima methods on GNNs. We recommend practitioners to always use weight averaging techniques, in particular EWA when using early stopping. While weight averaging techniques are only sometimes the best performing method, they are less sensitive to hyperparameters, need no additional training, and keep the original model unchanged. All source code is available under https://github.com/Foisunt/FMMs-in-GNNs.
BibTeX
@inproceedings{lell2023split,
  author = {Nicolas Lell and Ansgar Scherp},
  title = {{The Split Matters: Flat Minima Methods for Improving the Performance of GNNs}},
  booktitle = {CD-MAKE},
  year = {2023},
  url = {https://openreview.net/forum?id=VTQaEf21TV},
}
Figure