In the world of deep learning, "deeper is better" is a long-standing mantra. However, Graph Neural Networks (GNNs) have historically been the exception. While ResNets and Transformers scale to hundreds of layers, standard GNNs often peak at just two or three. Adding more layers typically leads to over-smoothing, a phenomenon where node representations converge to the same value, making them indistinguishable and rendering the network useless.
Recent research has fundamentally challenged this "shallow-only" constraint. By rethinking message passing as a dynamical system, introducing structural sparsity, and preserving initial signals, researchers are now training GNNs with dozens, hundreds, or even a thousand layers.
The Over-smoothing Problem as Signal Collapse
To understand why GNNs fail at depth, consider the basic graph convolution: a node updates its state by averaging the features of its neighbors. In a shallow network, this captures local context. But as you stack layers, each node's "receptive field" expands to cover almost the entire graph.
Yulong Yang and colleagues in Understanding Oversmoothing in GNNs as Consensus in Opinion Dynamics frame this as a "consensus" problem. Just as repeated averaging in a social group leads everyone to hold the same opinion, repeated graph convolutions lead all node embeddings to a uniform state. To build deep GNNs, we must prevent this collapse.
Preserving the Identity: Residuals and Mapping
One of the most effective ways to combat over-smoothing is to ensure the network never "forgets" the original input features. Ming Chen and Zhewei Wei introduced GCNII in Simple and Deep Graph Convolutional Networks, which uses two key mechanisms:
- Initial Residual Connections: Instead of just adding a skip connection from the previous layer, every layer receives a fraction of the original input features.
- Identity Mapping: Adding an identity matrix to the weight matrix ensures that, even with deep stacks, the model can easily learn to pass information through without transforming it into noise.
This approach allows GNNs to scale to 64 layers and beyond. Similarly, Guosheng Feng and Chunnan Wang used these principles in Search For Deep Graph Neural Networks to automatically discover deep architectures that outperform hand-crafted shallow models.
GNNs as Physical Systems
A powerful shift in perspective treats the layers of a GNN not as discrete steps, but as a continuous evolution of a physical system.
In Graph-Coupled Oscillator Networks, T. Konstantin Rusch and Michael M. Bronstein model node features as a system of coupled oscillators governed by second-order ordinary differential equations (ODEs). This physical grounding naturally prevents over-smoothing because the system's energy is preserved rather than dissipated.
Similarly, Alessio Gravina and Davide Bacciu proposed Anti-Symmetric DGN in Anti-Symmetric DGN: a stable architecture for Deep Graph Networks. By constraining weight matrices to be anti-symmetric, they ensure the network's hidden states remain stable across dozens of layers, effectively solving the "forgetting" problem in deep architectures.
Breaking the Consensus with Sparsity and Noise
If over-smoothing is caused by too much communication, one solution is to simply talk less.
Yu Rong and Wenbing Huang introduced DropEdge in The Truly Deep Graph Convolutional Networks for Node Classification. By randomly removing edges during training, they reduce the "message-passing" connectivity, which delays the convergence of node features.
More advanced pruning strategies like ClusterDrop, proposed by Wenjing Lu and Yang Yang in An Efficient Prototype-Based Clustering Approach for Edge Pruning, use clustering to identify and remove "inter-class" edges that cause different communities to blur together. This keeps the boundaries between different types of nodes sharp, even at extreme depths.
Adaptive and Node-Specific Depth
Not every node in a graph needs the same amount of context. A node in a dense cluster might only need two layers of information, while a node on a sparse periphery might need twenty.
Teng Xiao and Donglin Wang addressed this in Learning How to Propagate Messages in Graph Neural Networks by treating the number of propagation steps as a latent variable. Their L2P framework learns a personalized depth for every node.
In a similar vein, Hao Wu and Guohao Li developed NoSAF (All Nodes are created Not Equal), which uses node-specific filters to selectively retain or replenish information at each layer. By allowing the network to "gate" how much information flows into each node, these models can scale to 64 or 100 layers without the typical performance collapse.
Scaling to the Extreme
The frontier of this research is pushing GNNs to depths previously thought impossible. Gradient Gating ($G^2$), described in Gradient Gating for Deep Multi-Rate Learning on Graphs, uses local graph gradients to modulate updates. This mechanism allows for the training of GNNs with up to 1,000 layers, proving that when over-smoothing is properly managed, depth becomes a powerful asset for capturing complex, multi-scale graph structures.