Scaling Graph Neural Networks to Massive Datasets

This direction is merged into another direction: graph neural network scalability (graph memory scalability). Its last version stays readable and citable here.

Version 1 of 141 papers43 conceptsfrozen 2026-10-04

The fundamental challenge of Graph Neural Networks (GNNs) is a spatial one. Unlike images or text, where data points are independent or locally constrained, graphs are defined by their connectivity. In the seminal work by Thomas Kipf and Max Welling, Semi-Supervised Classification with Graph Convolutional Networks established the "full-batch" training paradigm. This requires the entire adjacency matrix and all node features to reside in GPU memory simultaneously.

While effective for small citation networks, this approach hits a "memory wall" when applied to web-scale graphs. As researchers like Jianfei Chen and Le Song point out in Stochastic Training of Graph Convolutional Networks, a single node’s receptive field grows exponentially with the number of layers—a phenomenon known as "neighborhood expansion." To make GNNs practical for billions of edges, the research community has shifted toward breaking this structural coupling.

The Sampling Solution

The most direct way to bypass the memory wall is to stop looking at the whole graph at once. Xin Liu and colleagues provide a taxonomy of these efforts in Sampling methods for efficient training of graph convolutional networks: A survey.

The intuition is simple: if we can't fit the ocean in a bucket, we take a cup of water. However, naive sampling introduces high variance and bias. To solve this, Jianfei Chen introduced control variates, which use historical activations to "fill in the gaps" of unsampled neighbors. This allows a model to achieve full-batch accuracy while only sampling two neighbors per node. Other approaches, like the Text Graph Transformer by Haopeng Zhang, use PageRank-based sampling to ensure that the most structurally important nodes are preserved in every mini-batch.

Decoupling and Layer-wise Training

If sampling addresses the "width" of the neighborhood expansion, layer-wise training addresses the "depth." In L2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networks, Yuning You and Zhangyang Wang propose decoupling feature aggregation from feature transformation. By processing the graph layer-by-layer and freezing previous states, the memory footprint remains constant regardless of how deep the network goes.

A more radical form of decoupling is found in Node2Grids by Chuan Chen. Instead of performing convolutions on the graph itself, the method maps neighborhoods into independent, grid-like structures. This transforms a graph problem into a computer vision problem, allowing researchers to use standard, highly optimized CNN mini-batching techniques that don't require the global graph structure to be in memory.

Multiscale and Coarsened Representations

Another recurring strategy for efficiency is graph coarsening. The idea is to solve a simpler version of the problem first. Manoj Kumar and Sandeep Kumar demonstrate in Q-MAGC that by grouping similar nodes into "super-nodes," one can perform clustering much faster without losing theoretical consistency.

Similarly, Eli Gal and Moshe Eliasof propose a Multiscale Approach where the model trains on progressively finer versions of the graph. This "zoom-in" strategy ensures that the heavy lifting of learning representations happens on small, manageable data, with the full-scale graph only used for final refinement.

Distributed Systems and Hardware Co-design

When a graph is so large that it cannot fit on a single machine even with sampling, the problem becomes one of communication. Hesham Mostafa introduced Sequential Aggregation and Rematerialization (SAR) in this paper to enable full-batch training across a cluster. SAR reconstructs parts of the computational graph on the fly during the backward pass, freeing up memory and allowing training on graphs with billions of edges without the noise introduced by sampling.

Finally, researchers are looking at the hardware itself. Cheng Wan and the MixGCN team argue that scaling requires a "mixture of accelerators." By splitting sparse operations (like neighborhood aggregation) and dense operations (like weight updates) across different types of hardware, they minimize the communication bottlenecks that typically cripple distributed GNN training.

Through these diverse lenses—sampling, decoupling, coarsening, and hardware optimization—the field is moving away from the constraints of the full-batch adjacency matrix toward a more fluid, scalable architecture for relational data.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_graph_neural_networks_to_massive_4c39e58f,
  title = {Scaling Graph Neural Networks to Massive Datasets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-graph-neural-networks-to-massive-datasets/rdr_4c39e58ff0f9470f9d24c4db64f39861},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_graph_neural_networks_to_massive_4c39e58f_v1,
  title = {Scaling Graph Neural Networks to Massive Datasets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-graph-neural-networks-to-massive-datasets/rdr_4c39e58ff0f9470f9d24c4db64f39861/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}