Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification

Yuankai Luo, Lei Shi, Xiao-Ming Wu

TL;DR

This paper challenges the prevailing notion that Graph Transformers (GTs) universally outperform traditional Graph Neural Networks (GNNs) for node classification. By meticulously tuning hyperparameters of classic GNNs (GCN, GAT, GraphSAGE), the authors demonstrate that these foundational models can achieve state-of-the-art performance, often surpassing recent GTs across a diverse set of 18 datasets. This work calls for a more rigorous empirical evaluation in graph machine learning.

What Problem Does This Paper Address?

The field of graph machine learning has seen a surge in interest in Graph Transformers (GTs), which are often presented as superior alternatives to traditional message-passing Graph Neural Networks (GNNs) for tasks like node classification. GTs, with their self-attention mechanisms, are theoretically capable of capturing global dependencies, while GNNs primarily focus on local neighborhood information. Many recent studies report GTs significantly outperforming GNNs on various benchmarks. However, this paper questions whether this superiority is truly inherent or if it stems from suboptimal hyperparameter tuning of GNNs in previous evaluations. The authors aim to re-evaluate classic GNNs by applying careful hyperparameter optimization to see if they can indeed match or exceed the performance of modern GTs.

Methods

The core of this paper is a comprehensive empirical re-evaluation of three classic GNN models: Graph Convolutional Networks (GCN) [28], Graph Attention Networks (GAT) [68], and GraphSAGE [20]. The authors meticulously tune a range of key hyperparameters for these models, including:

  • Normalization: Layer Normalization (LN) or Batch Normalization (BN) applied before the activation function.
  • Dropout: A regularization technique applied to feature embeddings.
  • Residual Connections: Used to alleviate vanishing gradients and improve information flow.
  • Network Depth: The number of GNN layers, explored from 2 up to 10 (and beyond for specific heterophilous graphs).
  • Optimizer: Adam optimizer with a learning rate search space.
  • Hidden Dimension: The dimensionality of node embeddings.

They also investigate the impact of these hyperparameters through ablation studies. For comparison, they evaluate against several state-of-the-art Graph Transformers.

Datasets

The study utilizes a diverse set of 18 datasets, categorized into:

  • Homophilous Graphs: Cora, CiteSeer, PubMed, Computer, Photo, CS, Physics, WikiCS. These datasets are characterized by edges connecting nodes with similar attributes or labels.
  • Heterophilous Graphs: Squirrel, Chameleon, Roman-Empire, Amazon-Ratings, Minesweeper, Questions. These datasets feature edges that often connect nodes with different attributes or labels, posing a greater challenge for traditional GNNs.
  • Large-scale Graphs: ogbn-arxiv, ogbn-proteins, ogbn-products (from the Open Graph Benchmark), and pokec. These datasets involve millions of nodes and edges, requiring efficient training strategies.

Key Insights and Ideas

  • GT Superiority May Be Overstated: The most significant finding is that with proper hyperparameter tuning, classic GNNs can achieve state-of-the-art performance, often matching or exceeding that of recent Graph Transformers. This suggests that previous comparisons might have been biased by suboptimal GNN configurations.
  • Normalization is Crucial for Large Graphs: While less impactful on smaller graphs, normalization techniques (BN/LN) become essential for maintaining stable training and achieving high accuracy on large-scale graphs, likely due to the wider variety of node features and resulting data distributions.
  • Dropout is Consistently Essential: The ablation of dropout consistently leads to notable accuracy declines across all graph types, highlighting its critical role in preventing overfitting and maintaining performance.
  • Residual Connections Boost Heterophilous Performance: Residual connections significantly improve performance, especially on heterophilous graphs. This is attributed to their ability to help capture complex relationships in these graphs, which often require deeper architectures.
  • Deeper Networks Benefit Heterophilous Graphs More: While deeper networks (more layers) generally improve performance on heterophilous graphs compared to homophilous ones, the study suggests that extremely deep GNNs (beyond 10-15 layers) may not offer substantial gains and can even lead to performance degradation.

Key Results

  • Dominance on 17 out of 18 Datasets: The tuned classic GNNs (GCN, GraphSAGE, GAT*) achieved the top rank on 17 out of the 18 evaluated datasets, demonstrating their robust competitiveness against advanced GTs.
  • Significant Accuracy Improvements: For instance, on the WikiCS dataset, GAT saw a 4.16% accuracy increase, moving it to first place. On Roman-Empire, GCN performance improved by 17.58%.
  • Large-Scale Graph Performance: On large-scale datasets like ogbn-arxiv and pokec, GCN* emerged as the leading model. For pokec, all three classic GNNs achieved over 10% performance increases with the authors' implementation.
  • Visualization Insights: t-SNE visualizations showed that the node embeddings generated by the tuned GCN exhibited greater inter-class distances compared to those produced by Polynormer, suggesting better class separability.

Limitations and Future Work

  • Focus on Node Classification: The study is limited to the node classification task. The authors suggest extending their benchmarking efforts to include classic GNNs in graph-level and edge-level tasks, as well as graph classification and link prediction.
  • Hyperparameter Tuning Space: While extensive, the hyperparameter search space might not cover all possibilities. The authors mention that their implementation is available for further exploration.
  • Jumping Knowledge (JK) Mode: Initially considered a hyperparameter, the JK mode was found to be not consistently necessary, with results often comparable or better without it after thorough tuning. This suggests that careful tuning of other hyperparameters can mitigate the need for such complex mechanisms.

Summary updated

Abstract

Graph Transformers (GTs) have recently emerged as popular alternatives to traditional message-passing Graph Neural Networks (GNNs), due to their theoretically superior expressiveness and impressive performance reported on standard node classification benchmarks, often significantly outperforming GNNs. In this paper, we conduct a thorough empirical analysis to reevaluate the performance of three classic GNN models (GCN, GAT, and GraphSAGE) against GTs. Our findings suggest that the previously reported superiority of GTs may have been overstated due to suboptimal hyperparameter configurations in GNNs. Remarkably, with slight hyperparameter tuning, these classic GNN models achieve state-of-the-art performance, matching or even exceeding that of recent GTs across 17 out of the 18 diverse datasets examined. Additionally, we conduct detailed ablation studies to investigate the influence of various GNN configurations, such as normalization, dropout, residual connections, network depth, and jumping knowledge mode, on node classification performance. Our study aims to promote a higher standard of empirical rigor in the field of graph machine learning, encouraging more accurate comparisons and evaluations of model capabilities.
BibTeX
@inproceedings{luo2024classic,
  author = {Yuankai Luo and Lei Shi and Xiao-Ming Wu},
  title = {{Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure