Graph neural networks (GNNs) update each node by averaging features from its neighbors. When many such layers are stacked, the averaging erases the differences between nodes, a failure called over-smoothing. The two standard fixes both reuse earlier layers. Jumping Knowledge networks hand every intermediate layer to the final classifier. GCNII adds the original input back into every layer. Both fixes sum or concatenate features that have been through very different amounts of averaging, and neither checks whether those features are on comparable scales first.
Computer vision learned long ago that multi-depth features must be normalized before they are fused. The obvious transplant is to put a LayerNorm or BatchNorm on each stream before fusion. We argue that this transplant normalizes the wrong quantity on graphs. A deep graph layer is mostly one known direction, the stationary component that repeated averaging converges to. The node-distinguishing signal is a small remainder, and it shrinks faster at high-degree nodes. We propose to split each stream at the fusion point into that collapse component and the remainder. The remainder is rescaled to a common energy, computed separately for each degree group. The collapse component is kept with its own learned weight, so nothing is discarded.
Core Claim
Take the node-feature matrix $H^{(\ell)}$ at layer $\ell$ and the symmetric normalized propagation matrix $\tilde P = D^{-1/2}(A+I)D^{-1/2}$. Repeated application of $\tilde P$ drives features toward the direction $u = D^{1/2}\mathbf{1}/\lVert D^{1/2}\mathbf{1}\rVert$, where $D$ is the degree matrix including self-loops. Before a stream enters a fusion point, we write $H^{(\ell)} = u u^\top H^{(\ell)} + R^{(\ell)}$ and replace it with
$$\hat H^{(\ell)} = a_\ell\, u u^\top H^{(\ell)} + b_\ell\, S_\ell^{-1} R^{(\ell)},$$
The pieces of this formula are as follows. $R^{(\ell)}$ is the detail component orthogonal to $u$. $a_\ell$ and $b_\ell$ are learned scalars. $S_\ell$ is a diagonal matrix. Its entry for node $i$ is the root-mean-square norm of the rows of $R^{(\ell)}$ belonging to nodes in $i$'s degree bucket.
The transform is invertible and scale-only on each component. No signal is projected away, which respects the finding that mean-centering is what destroys community structure. The transform is applied only at fusion points: before the Jumping Knowledge readout, and to $\tilde P H^{(\ell)}$ just before GCNII adds $H^{(0)}$. The propagation path between layers is left unchanged.
The claim is falsifiable in a precise way. If per-stream LayerNorm or learned fusion weights match this transform at depth, the claim is wrong.
Two arguments support it. First, total-magnitude normalization is dominated by the collapse component in deep streams, so it cannot restore the detail energy that is actually shrinking. Second, a learned linear readout or fusion weight can undo one global scale per layer. It cannot undo a mismatch that differs by node degree.
Where This Sits in Prior Work
The vision literature supplies the method template. Inside-Outside Net applies L2 normalization with a learned scale to features from different depths before concatenating them, which stabilized training. Multilevel Context Representation for Improving Object Recognition shows that the highest-magnitude layer dominates gradients and classifier weights, which motivates the concern here.
BA-Net and ResNet or DenseNet? Introducing Dense Shortcuts to ResNet take a similar route, applying BatchNorm or GroupNorm to each branch before summation. Those normalizers subtract means, and that is exactly what graph theory warns against.
The warning comes from Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs. It proves that dividing by the standard deviation prevents collapse, while mean-centering damages community eigenvectors, and it replaces centering with a learned projection (GraphNormv2) inside each layer. GRANOLA reports a related effect: centering pushes degree signals negative, where ReLU zeroes them. These two papers constrain our design and are why our transform only rescales. We differ from both by acting across layers at fusion points, not within a layer.
The existing graph normalizers operate inside each layer on the forward path, and none of them treats degree as the axis along which scale should be equalized. PairNorm subtracts the uniform node mean and then fixes the total pairwise distance. ContraNorm applies a repulsive update against dimensional collapse, and explicitly leaves open how normalization should interact with skip connections. Differentiable Group Normalization normalizes within learned soft clusters at a single depth.
For GCN-style propagation the direction of collapse is $D^{1/2}\mathbf{1}$, not the uniform vector. PairNorm's uniform centering therefore both removes some useful signal and leaves some collapse in place.
The fusion architectures are Jumping Knowledge Networks, which concatenate, max-pool, or attend over all layers without normalization, and GCNII, which mixes $\tilde P H^{(\ell)}$ and $H^{(0)}$ with a fixed coefficient. Both papers also supply the key motivating fact: hub nodes mix faster than peripheral nodes. That is why our scale statistics are computed per degree bucket.
Finally, A Survey on Oversmoothing in Graph Neural Networks supplies a cautionary result. Keeping features far apart, even with noise, does not keep them useful. Our detail rescaling could amplify noise in exactly this way, so the experiments include the controls the survey implies.
Research Program
The work begins with measurement, before any new module is built. We will train standard GCNII models and Jumping Knowledge GCN and GraphSAGE models at depths from 2 to 64 on Cora, Citeseer and Pubmed. For every fusion point we will record three quantities, all broken down by degree quintile:
- the ratio of detail energy to collapse energy in each stream;
- the ratio of each stream's detail energy to that of $H^{(0)}$;
- per-layer gradient norms, since Backward Oversmoothing argues that training difficulty in deep GNNs lives partly in the gradients.
The premise holds only if deep streams show detail energy that is orders of magnitude below shallow streams, with a clear gap between high-degree and low-degree nodes. If the mismatch is small or uniform across degree, we stop and report that negative finding.
If the premise holds, we implement the transform as a drop-in module for the Jumping Knowledge readout and for GCNII's initial-residual sum. For row-normalized operators such as GraphSAGE and GAT, the collapse direction is the constant vector. For GAT the operator changes during training, so we estimate the direction with a few power iterations on each layer's attention matrix. GAT is treated as a stress test, because GraphNormv2 also gained less there.
For large graphs trained in mini-batches, the bucket statistics become running averages, as in BatchNorm. A small detached floor on $S_\ell$ bounds how much the transform can amplify the detail component, which is the obvious route to noise amplification.
We will also check a simple mechanistic prediction. For a linear GCN, the detail energy of layer $\ell$ decays roughly as $\lambda_2^{2\ell}$, where $\lambda_2$ is the second-largest eigenvalue magnitude of $\tilde P$. This form comes from the spectral decomposition of $\tilde P^\ell$. The decay should be faster in denser degree buckets. The learned $b_\ell$ should approximately invert this decay, and we will test whether it does.
Baselines
The naive reference points are GCNII and Jumping Knowledge exactly as published. For Jumping Knowledge we run all three of its fusion rules. The file gives Jumping Knowledge with max-pooling at 89.6% accuracy on Cora against 88.2% for a two-layer GCN, 0.818 micro-F1 on PPI with LSTM attention, and 0.965 micro-F1 on Reddit, all from Jumping Knowledge Networks. For GCNII's 64-layer Cora model, no reported result was found on Lacuna, so we reproduce it with the published hyperparameters.
The most direct competitor is the naive transplant from vision. It places a per-stream LayerNorm, BatchNorm, or L2 normalization with a learned scale (following Inside-Outside Net) at exactly the same fusion points. It is the strongest competitor on the core question, because it tests whether graph-specific structure matters at all.
The strongest existing graph method is GraphNormv2 from Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs. The file reports approximately 85.9% on Cora for GraphSAGE with GraphNormv2, against about 30–32% with BatchNorm or GraphNorm. No numeric result on ogbn-arxiv was found on Lacuna, only a qualitative statement that it significantly beats a failed 20-layer GCN.
We also include three intra-layer methods: PairNorm, ContraNorm (about 46% against about 19% for a vanilla 32-layer GCN on Cora, per the ContraNorm paper), and DGN (76.3% on Cora with missing features at 20 layers, against 74.5% for PairNorm with SGC, from their respective papers).
Three ablations isolate the key ingredient:
- one global detail scale per layer instead of per-degree buckets;
- projecting out the uniform mean instead of $D^{1/2}\mathbf{1}$;
- dropping the collapse component entirely, which amounts to centering, to test the prediction that dropping it hurts.
A further control replaces $u$ with a random unit vector.
All methods share the same backbone, depth, hidden width, data splits, and training budget. Every method, including ours, receives the same tuning budget: the recipe from Classic GNNs are Strong Baselines plus an identical random search. Results are averaged over ten seeds.
To count as beating the baselines, our method must beat both GraphNormv2 and the best per-stream standard normalizer. It must do so by more than the seed standard deviation, at depths of 32 and above, on at least the missing-features setting and ogbn-arxiv, while not losing at shallow depth.
Why not just…?
Why not just put LayerNorm or BatchNorm on each stream before fusion? This is the real threat, and it may work. Our argument is that in a deep stream the total norm is dominated by the collapse component. Dividing by that norm therefore leaves the detail component as tiny as before. The measurement phase quantifies this directly, and the matched-placement comparison settles it. If per-stream LayerNorm matches us at depth 64, the proposal collapses to "use LayerNorm at fusion points". We would report that, and it would still be a new and useful finding for Jumping Knowledge networks and GCNII.
Why not just learn the fusion weights, for example a learned coefficient in GCNII or attention in Jumping Knowledge? A learned scalar per layer, or a linear readout over concatenated features, can undo a single global scale. That is why scale mismatch in plain concatenation matters mostly through optimization and weight decay. What a learned scalar cannot undo is a per-node mismatch that depends on degree. Jumping Knowledge's LSTM attention is computed per node, so it could in principle learn degree-dependent weights. It is therefore included as a baseline, and the global-scale ablation of our own method tests whether degree conditioning adds anything beyond learned weights.
Why not just use a good intra-layer normalizer such as GraphNormv2 or PairNorm? Intra-layer scaling already keeps features from collapsing in rank. However, the survey shows that PairNorm's accuracy on Cora still falls beyond 32 layers even though pairwise distance is maintained. GraphNormv2 itself was weaker on GAT. We run both, and we also run each one combined with our module. Combining would be pointless if intra-layer scaling already equalized scales across layers, so a gain from the combination directly answers this objection.
Why not just use a shallow, well-tuned GNN? On standard dense-feature citation benchmarks, a shallow model is probably enough, as Classic GNNs are Strong Baselines shows. The case for depth comes from settings where information must travel many hops. With missing features on Cora, a GCN peaks at 3 layers with 57.3% accuracy, while deep normalized models reach 74.5–76.3%. We report tuned two- and three-layer models alongside every deep result, and we concede the gains on dense-feature benchmarks if they fail to appear there.
Decisive Experiments
The first decisive check is the diagnostic itself. The detail-energy gap between layers must exist and must depend on degree in trained GCNII and Jumping Knowledge models.
The second is the missing-features setting on Cora, Citeseer and Pubmed at 8 to 64 layers, where depth is actually needed. Here we must beat per-stream LayerNorm, GraphNormv2 and DGN's 76.3%.
The third is ogbn-arxiv at 20 or more layers, using the realistic time-based split from Open Graph Benchmark. We would add ogbn-products as a mini-batch scalability check against GraphSAGE's 78.50%.
The fourth is PPI and Reddit with Jumping Knowledge fusion, measured against 0.818 and 0.965 micro-F1 respectively. Wisconsin tests whether rescaling the detail component helps or hurts on a heterophilic graph, where neighbors tend to differ.
For every run we report per-degree-quintile accuracy. The mechanism predicts that the gains should concentrate on high-degree nodes. A gain that is uniform across degrees, or one that the random-direction control also reproduces, would mean the improvement is not coming from the proposed mechanism.
References
- Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
- Multilevel Context Representation for Improving Object Recognition
- BA-Net: Bridge Attention for Deep Convolutional Neural Networks
- ResNet or DenseNet? Introducing Dense Shortcuts to ResNet
- Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs
- GRANOLA: Adaptive Normalization for Graph Neural Networks
- PairNorm: Tackling Oversmoothing in GNNs
- ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond
- Towards Deeper Graph Neural Networks with Differentiable Group Normalization
- Representation Learning on Graphs with Jumping Knowledge Networks
- Simple and Deep Graph Convolutional Networks
- A Survey on Oversmoothing in Graph Neural Networks
- Backward Oversmoothing: why is it hard to train deep Graph Neural Networks?
- Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification
- Open Graph Benchmark: Datasets for Machine Learning on Graphs