Standard neural network initialization methods like Xavier or Kaiming assume that data points are independent and identically distributed. In the world of Graph Neural Networks (GNNs), this assumption collapses. Because GNNs rely on message-passing—where a node’s representation is a function of its neighbors—the variance of a signal is dictated by the graph's connectivity. Research across the field now shows that "clogged" training dynamics and the infamous "oversmoothing" problem are often not architectural failures, but initialization failures.
The Variance-Shifting Effect of Graph Topology
In a standard MLP, the variance of a layer's output depends on the layer's width. In a GNN, the variance depends on the graph's adjacency matrix. As Sitao Luan and Jiahang Li observe, the aggregation of neighbor features acts as a variance-shifting operation. If you use a standard initializer, the signal magnitude can explode or vanish exponentially based on the average node degree.
To fix this, researchers have moved toward "topology-aware" initialization. The core intuition is to scale the initial weight variance by the inverse of the node degrees or the graph's spectral properties. For example, Dimitrios Kelesis and Georgios Paliouras demonstrate in their work on G-Init that incorporating node degrees into the initialization logic allows GNNs to maintain distinct node representations at depths of 64 layers, far beyond the typical 2-3 layer "ceiling."
Preventing Oversmoothing via Signal Stability
Oversmoothing occurs when node features converge into a uniform blur, effectively turning the GNN into a low-pass filter. Bastian Epping and Michael T. Schaub prove that by increasing initial weight variance, GNNs can enter a "chaotic" phase where features remain distinct even in networks with hundreds of layers.
Similarly, AJAY KUMAR JAISWAL and Tianlong Chen show that "vanilla" GCNs can outperform complex state-of-the-art models simply by using an isometric initialization that preserves the geometry of node features across layers. By ensuring the transformation is nearly distance-preserving, the model prevents the feature collapse that usually plagues deep graph architectures.
Generalizing to Complex Architectures
This principle extends beyond simple GCNs to more exotic architectures:
- Tensorial Networks: Ao Liu shows that in compressed models using tensor decompositions, the variance depends on the number of internal "contracting edges." Standard methods ignore this internal topology, leading to unstable training.
- Directed Acyclic Graphs (DAGs): Libin Zhu and Misha Belkin generalize this by scaling initialization by the inverse square root of a neuron's specific in-degree, ensuring stable gradients in any feedforward network regardless of how irregular its branching is.
- Automated Search: Senmiao Wang and Yushun Zhang propose SPoGInit, which treats initialization as a pre-training optimization problem. By using zeroth-order optimization to find the "ideal" variance for a specific graph's topology, they enable stable signal propagation in extremely deep models.
By shifting from "one-size-fits-all" weights to structural awareness, these methods allow GNNs to scale in depth and complexity, proving that the limitations of graph learning are often a matter of how the journey begins.