Inhibitory Feedback on Gradients for Training Deep Graph Networks

Graph neural networks (GNNs) are models that update each node's features by averaging its neighbors' features, layer after layer. When many such layers are stacked, accuracy collapses. The usual explanation is over-smoothing: repeated averaging pushes every node toward the same representation. A newer explanation says the backward pass is the problem. The error signal sent back to each node also gets averaged, so it becomes the same at every node and stops telling earlier layers anything useful. This note proposes to test which explanation is right by intervening on the backward pass alone. The intervention borrows a mechanism from cortex. There, recurrent inhibitory feedback senses the slow, shared component of a neural population's activity and cancels it, which leaves the population's independent fluctuations free to carry information. We apply the same kind of slow negative feedback to the per-node error signals during training, and we leave the forward computation untouched.

The optimizer is central to this question, and prior work has ignored it. Adam rescales each coordinate of the gradient by its recent magnitude. On the backward-collapse picture, that rescaling has two possible effects. It could rescue deep GNNs, because it inflates vanishing gradients. Or it could trap them, because it spends the inflated step entirely on the one uninformative direction that all nodes share. We predict the second. If the prediction holds, it would explain part of the gap between adaptive optimizers and plain stochastic gradient descent (SGD) in deep message passing. It would also give a cheap fix that does not require the very wide layers or forward-pass normalization that current remedies use.

Core Claim

In a deep GNN, the gradient of the loss with respect to layer $l$'s node features is $G_l \in \mathbb{R}^{n\times d}$, with $n$ nodes and $d$ channels. Let $u$ be the dominant eigenvector of the normalized adjacency (proportional to the square root of node degree). We define the sum-mode fraction

$$s_l = \frac{\|u^\top G_l\|^2}{\|G_l\|_F^2},$$

which is the share of the backward signal that is identical in shape across all nodes.

The claim has three parts. First, $s_l$ rises with depth before test accuracy collapses. Second, it rises faster under Adam than under SGD with momentum. Third, removing only the slowly varying part of this component from the backward signal restores trainability at depths of 32 to 64 layers, even if forward feature rank is left to decay.

The intervention keeps a running estimate $m_l \leftarrow (1-\tau)m_l + \tau\, u^\top G_l$ of the shared error component, where $\tau$ is a small averaging rate that sets the feedback's time constant in training steps. Before the signal is passed to the next layer down and into the optimizer's moment estimates, the intervention replaces it as follows:

$$G_l \leftarrow G_l - \kappa_l\, u\, m_l, \qquad \kappa_l \leftarrow \kappa_l + \eta\,(s_l - s^\ast).$$

Here $\kappa_l$ is an inhibitory gain that is adapted homeostatically toward a target sum-mode fraction $s^\ast$ with a rate $\eta$.

The slow running average is the biologically motivated ingredient. Inhibitory feedback suppresses population fluctuations by more than three orders of magnitude only at low frequencies and lets fast independent fluctuations pass (Decorrelation of Neural-Network Activity by Inhibitory Feedback, Decorrelation of low-frequency neural activity by inhibitory feedback). In this setting, suppressing only the slow part removes the persistent shared drift and keeps the step-to-step mini-batch noise. That noise is argued to help SGD escape poor minima (Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning). The cancellation also emerges as a stability requirement on the population's sum mode rather than as pairwise tuning (The Correlation Structure of Local Neuronal Networks Intrinsically Results from Recurrent Dynamics). That is why we target a single global mode and do not attempt full gradient whitening.

Why This Is Not Already Done

Several lines of work motivate the proposal. Backward Oversmoothing proves that backpropagated signals in deep GNNs become constant across nodes at an exponential rate. In that analysis the last layer converges early and acts as a "gatekeeper," stranding earlier layers at high-loss stationary points. The paper analyzes plain gradient flow and explicitly leaves open whether alternative optimizers can steer around these traps. We take up that question directly.

Two papers supply the measurement tools and the optimizer hypothesis. Rethinking Oversmoothing in Graph Neural Networks: A Rank-Based Perspective shows that effective rank tracks accuracy far better than Dirichlet energy, with correlation 0.9103 versus −0.7179. We reuse its rank metrics for the forward pass and add $s_l$ for the backward pass. Pre-Normalization Momentum and Rank Bias in Adaptive Optimizers shows that Adam's momentum acts as a temporal low-pass filter that biases solutions toward low rank. In deep factorization it recovers rank 4.0, where RMSProp overshoots to 22.3. That paper treats low-rank bias as a benefit and never considers graphs. Our hypothesis is that in GNNs the same filter compounds the shared-mode collapse. The inhibitory feedback is, in that sense, a temporal high-pass counterpart placed on the signal before it enters the optimizer.

Existing remedies all act on the forward pass, which constrains the design. PairNorm centers and rescales features to keep pairwise distances fixed. The Mean-Subtraction trick in Revisiting "Over-smoothing" in Deep GCNs subtracts channel means before convolution. Filterbank and scattering designs add fixed high-pass channels (Complete the Missing Half, Geometric Scattering Attention Networks). Partially trained graph convolutional networks resist oversmoothing uses frozen random layers of width 8,192 as static decorrelators, at a large memory cost.

Forward centering has a consequence that matters here. Its Jacobian is itself a projection, so it also removes a uniform component of the backward signal at the point where it is inserted. Forward-pass remedies therefore confound the forward and backward explanations. Our design separates the two by changing only the backward signal. It targets the degree-weighted mode $u$ rather than the uniform vector. It is time-filtered rather than instantaneous. It costs $O(nd)$ per layer and adds no parameters.

Two papers contradict the proposal, and the experiments engage both. Revisiting "Over-smoothing" in Deep GCNs attributes the drop in deep GCNs to overfitting rather than collapse. The Oversmoothing Fallacy attributes it to vanishing signals from the combination of linear maps and nonlinearities, and shows that residual connections with BatchNorm reach 1,024 layers. Our measurements report train loss, test loss, and $s_l$ together for every run, so each of the three accounts makes a distinguishable prediction.

Research Program

The first stage is diagnostic. We train plain deep graph convolutional networks (GCNs) in the formulation of Semi-Supervised Classification with Graph Convolutional Networks at depths 2, 4, 8, 16, 32, and 64. We use SGD with momentum, Adam, RMSProp, and Padam with $p \in \{1/8, 1/4\}$ (Closing the Generalization Gap of Adaptive Gradient Methods), which interpolates between SGD and Adam. For every run we log three things over training. The first is the per-layer sum-mode fraction $s_l$. The second is the forward effective rank and numerical rank of the features. The third is the effective rank of each layer's accumulated weight change.

The prediction is that Adam drives $s_l$ toward 1 in early layers sooner than SGD does. We also predict that Adam's update magnitude stays large while its update rank collapses, and that $s_l$ predicts final accuracy beyond what forward effective rank already explains. We test the last point with partial correlation across runs.

The second stage applies the inhibitory feedback under each optimizer. Two decoupling ablations are central. In the first, the feedback runs on the backward pass with no forward change. In the second, forward centering runs with a stop-gradient, so the forward pass is centered but the backward signal is untouched. If the backward-only version recovers deep-network accuracy and the forward-only version does not, backward collapse is causal. The reverse outcome would falsify our hypothesis and support the classic forward account. We also run instantaneous projection ($\tau = 1$, $\kappa = 1$) against slow feedback, to test whether keeping the fast fluctuations matters. Finally, we place the feedback before versus after the second-moment estimate, to test whether the benefit comes from changing what Adam's preconditioner measures.

The third stage tests the result under the strongest standard architecture: residual connections with BatchNorm, pushed to 128 layers.

Baselines

The naive reference point is a plain deep GCN trained with tuned Adam, together with the 2-layer GCN. The cited material reports 81.5% accuracy on Cora for the 2-layer GCN (Semi-Supervised Classification with Graph Convolutional Networks) and 10–20% for a 32-layer standard GCN (Partially trained graph convolutional networks resist oversmoothing).

The strongest forward-pass competitor is Mean-Subtraction, because it is parameter-free and incidentally centers the backward signal. Revisiting "Over-smoothing" in Deep GCNs reports about 74.1% on Cora at 64 layers for this method, ahead of BatchNorm at about 73.9% and PairNorm at about 72.2%. We include PairNorm as well, because the PairNorm paper shows it remains strong when node features are missing.

The strongest architectural competitor is residual connections with BatchNorm from The Oversmoothing Fallacy. We found no reported accuracy number on Lacuna for it on the heterophilic graphs used here; that paper reports qualitatively that deep models outperform shallow ones on Chameleon. We also include partial training at reduced widths of 512 and 2,048 so it can be compared under matched memory. The source reports greater than 70% at 32 layers on Cora with width 8,192.

The optimizer baselines are tuned SGD with momentum and Padam. We found no reported result on Lacuna for any adaptive-versus-SGD comparison on GNNs, so we generate this comparison ourselves.

Fairness is enforced in four ways. Every method and optimizer pair receives the same random-search budget of 60 trials over learning rate, weight decay, dropout, and optimizer-specific constants; for our method these are $\tau$ and $s^\ast$. All methods use the same public splits, 10 seeds, the same epoch limit with validation-based early stopping, and the same hidden width. Beating Mean-Subtraction means the following: at depth 32 or 64, the backward-only feedback matches or exceeds Mean-Subtraction under Adam with non-overlapping 95% intervals on at least half the datasets, while adding no forward computation. The key ablation is backward-only feedback against stop-gradient forward centering. It is the one comparison that separates our mechanism from what existing normalizers already do.

Why not just…?

Why not just use residual connections and BatchNorm? This is the most serious threat. The Oversmoothing Fallacy trains 1,024-layer GCNs this way. Backward Oversmoothing says skip connections only partly mitigate backward collapse and are not fully understood. We will measure $s_l$ in residual-plus-BatchNorm networks. If $s_l$ stays low there and the feedback adds nothing, we will report that the method is redundant with that architecture. The proposal's lasting value would then be the diagnostic, which would explain why the residual-plus-BatchNorm recipe works. A gain on top of residual-plus-BatchNorm at 128 layers, especially on heterophilic graphs, would settle the question the other way.

Why not just subtract the mean in the forward pass? Mean-Subtraction already removes a uniform component from both passes and has the best reported deep-network number (about 74.1% at 64 layers on Cora). It may plausibly match our method on accuracy alone. The stop-gradient ablation is designed to answer this. If forward centering only helps through its backward Jacobian, then stop-gradient centering will fail and backward-only feedback will succeed. If forward centering helps even with the stop-gradient, the forward account wins. Mean-Subtraction and the feedback also differ in two testable ways. Mean-Subtraction targets the uniform vector rather than the degree-weighted mode, which matters on graphs with skewed degrees. And it removes the shared mode instantly, which also removes the fast fluctuations.

Why not just switch to SGD or Padam? Padam closes the adaptive gap on CIFAR-10, with 93.78% against Adam's 92.21%, and on ImageNet, with 74.04% against 69.52%. None of those papers test GNNs, though. And if backward collapse comes from the graph operator itself, as Keriven's analysis implies, SGD will also collapse at depth, only more slowly. The Padam sweep over $p$ tests this directly. A smooth decline in $s_l$ as $p$ decreases, with deep-network accuracy still collapsing at $p = 0$, would show that changing optimizers is not enough. If SGD alone recovers deep accuracy, the optimizer is the main culprit and the feedback is unnecessary. We will report that outcome if it occurs.

Why not just freeze wide random layers? Partial training works at width 8,192, but its authors note that the memory cost rules it out on large graphs. We compare at matched memory on OGB-Arxiv, where our method adds only one $d$-dimensional buffer per layer.

Proposed Experiments

We use Cora, Citeseer, Pubmed, CoauthorCS, Physics, and WikiCS, where Backward Oversmoothing confirms the stationary plateau. We add Chameleon and Squirrel as heterophilic tests, because there the low-frequency mode is least informative. OGB-Arxiv serves as the scale test. The metrics are test accuracy, the train/test loss gap, forward effective rank and numerical rank, the sum-mode fraction $s_l$, update rank, and peak GPU memory.

The decisive checks are three. First, does $s_l$ rise with depth, and faster under Adam than SGD, before accuracy falls? Does it add predictive power beyond forward effective rank? Second, does backward-only feedback recover accuracy at 32–64 layers while forward rank is still decaying, and does stop-gradient forward centering fail to do so? Third, does the Adam-versus-SGD gap in deep GNNs shrink once the shared error mode is suppressed, and does slow feedback beat instantaneous projection?

A negative answer to the second check would falsify the backward-collapse account in favor of forward collapse. A negative answer to the third would mean the optimizer is not where the interaction lives. Either result would sharpen the three-way debate the field has not resolved.