Pinning the Firing Threshold: Does Temporal Drift Cause Over-Smoothing in Deep Spiking Graph Networks?

Spiking graph neural networks pass binary "spikes" between graph nodes instead of real numbers. That makes them cheap to run on low-power neuromorphic chips, with reported energy savings of roughly 700x on one such chip (Spiking Graph Convolutional Networks). Every spiking graph network built so far is shallow, though: three layers, or a single propagation step done before any spiking happens. Continuous graph networks are known to fail when made deep. Repeatedly averaging each node with its neighbors makes all node representations look alike, a failure called over-smoothing. Spiking networks have a second, separate instability called temporal covariate shift. Each neuron carries a voltage-like state, the membrane potential, from one simulation step to the next. That leftover charge shifts the distribution of potentials over time, so whole populations drift away from the firing threshold and either go silent or fire constantly.

This proposal argues that in a deep spiking graph network these two problems are coupled, and that the coupling runs in an unexpected direction. Neighbor averaging shrinks the differences between nodes. A hard firing threshold placed exactly at the population's average potential does not care how small those differences are: a node fires if it sits above average and stays silent if below. The threshold therefore acts as a sign detector that can, in principle, keep node identities distinct at any depth. What destroys this protection is temporal drift. Carried-over charge moves the population average away from the threshold, and once every node sits on the same side, all nodes emit identical spikes. Our claim is that collapse in deep spiking graph networks is caused mainly by temporal drift of the population's common level relative to the threshold, not by spatial smoothing alone. A cheap fix follows from this: at each layer and time step, re-center the threshold on the population's current mean membrane potential. After training, this fix reduces to a fixed schedule of thresholds per layer, time step and channel, which neuromorphic hardware can run.

Core Claim

Write the membrane potential of node $i$ at layer $l$ and simulation step $t$ as $V_i(l,t) = m(l,t) + d_i(l,t)$. Here $m(l,t)$ is the mean over all nodes (the common mode) and $d_i(l,t)$ is the node's deviation from it (the differential mode). Neighbor averaging shrinks $d_i$ geometrically with depth; this is the low-pass behavior that PairNorm formalizes as a contraction of pairwise distances. Leftover charge from step $t-1$ moves $m(l,t)$. Guo et al. show that this residual-driven drift persists even when the inputs to a layer are normalized. With a fixed threshold $V_{th}$, a node's spike depends on $m(l,t)+d_i(l,t)-V_{th}$. Once the drift $|m - V_{th}|$ exceeds the shrinking spread of $d_i$, every node emits the same spike. Our first hypothesis is that the layer where accuracy collapses is predicted by the ratio of node spread to threshold offset, $\rho(l,t)=\mathrm{std}_i\, d_i(l,t)\,/\,|m(l,t)-V_{th}|$, much better than by the shrinkage of $d_i$ alone.

The intervention, which we call common-mode pinning, is to fire on

$$s_i(l,t) = \Theta\big(V_i(l,t) - \hat m(l,t) - b_l\big),$$

where $\Theta$ is the step function and $\hat m(l,t)$ is the per-channel mean membrane potential over nodes at the current step. This uses only present and past information. During training it is computed from the batch, and at inference it is a stored running estimate. The term $b_l$ is a small learnable offset per layer that controls how sparse the spikes are. Pinning does no variance rescaling, because the step function ignores scale. This is the property that should let the spike pattern survive the geometric shrinkage of $d_i$.

Pinning differs from existing threshold-aware normalizations in where and how it acts. tdBN (Going Deeper With Directly-Trained Larger Spiking Neural Networks) rescales layer inputs to variance $(\alpha V_{th})^2$ using statistics pooled over all time steps. TAB (TAB) normalizes inputs with statistics accumulated causally over steps $[1,t]$. Both act on the input current before it is integrated into the membrane potential. Pinning acts on the integrated membrane potential, which includes the carried-over charge that Guo et al. identify as the source of drift.

Position Relative to Prior Work

The spiking-graph literature motivates the question but leaves it untested. STFN normalizes membrane potentials over features and time inside spiking versions of GCN and GAT, reaching 80.7% on Cora against 81.5% for continuous GCN. It stops at three layers and pools statistics over the whole sequence, which is not causal. DRSGNN learns a threshold for each neuron and reaches 82.50±0.51% on Cora, but it propagates features once before spiking. It therefore never stacks message passing between spiking layers, and its thresholds are fixed over time. GSG avoids collapse through curved geometry, and Dy-SIGN uses skip channels on graphs whose structure changes over calendar time. Neither measures smoothing across depth or drift across simulation steps. Our proposal is the first to stack many message-passing layers in which spiking happens at every step and to ask what makes them collapse.

Two papers constrain the design. Are we measuring oversmoothing in graph neural networks correctly? shows that Dirichlet energy misses collapse in trained networks, while effective rank tracks accuracy with an average Pearson correlation of 0.91. We therefore use rank, not energy, as the primary collapse measure. The same paper warns that high rank is necessary but not sufficient, since noise also has high rank. This warning applies directly to us: a sign detector can preserve meaningless differences as faithfully as meaningful ones. SpikingGCN appears to contradict our claim. Its Chernoff-bound argument says spike trains approximate continuous graph convolution, which suggests spiking networks inherit continuous over-smoothing. We address this directly. The approximation holds for rate coding when the threshold is not tied to the population mean. When the threshold tracks the mean, the neuron behaves like a sign function, not a linear rate encoder, and the approximation argument no longer applies. Whether this holds in practice is exactly what the experiments test.

Beyond ReLU supplies the conceptual lens. It shows that monotone activations make message passing contractive, while non-monotone ones create stable fixed points that are not uniform across nodes. A step function centered on the population mean is monotone but invariant to scale. We ask whether that invariance alone is enough to escape contraction, or whether leak and reset are also needed.

Research Program

We will build a deep spiking graph convolutional network in which message passing over a normalized adjacency matrix happens between spiking layers at every simulation step. Its neurons are leaky integrate-and-fire neurons, meaning the membrane potential accumulates input, decays a little each step, fires when it crosses the threshold, and resets after firing. Depth ranges over 2, 4, 8, 16, 32 and 64 layers, and the number of simulation steps $T$ ranges over 2, 4 and 8.

The work has three parts. The first is measurement. At every layer and step we log the common mode $m(l,t)$, the node spread $\mathrm{std}_i\, d_i$, the ratio $\rho(l,t)$, the effective rank and numerical rank of the membrane potential matrix, the same ranks for each step's binary spike matrix and for the spike-count matrix summed over steps, Dirichlet energy, and firing rates. This yields the first map of rank collapse over both depth and time in a spiking graph network.

The second part is causal manipulation of temporal drift. We vary the leak factor and reset mode (hard reset versus subtracting the threshold), which change how much charge carries over between steps and therefore how strong the drift is. If drift is the cause of collapse, slowing the leak should move the collapse layer earlier in unpinned networks and have little effect in pinned ones.

The third part is the intervention and its ablations. We compare pinning on the membrane potential against pinning on the input current, causal per-step statistics against statistics pooled over the whole sequence, and the use of $b_l$ versus none. We also test the inference-time version, in which the stored statistics are folded into a fixed table of thresholds indexed by layer, step and channel. This version requires no computation across nodes at test time.

Baselines

The naive reference is the same deep spiking graph network with leaky integrate-and-fire neurons and no normalization. Its depth-accuracy curve defines collapse. A continuous GCN at matched depth gives the non-spiking reference: 81.5% on Cora at shallow depth, as reported in STFN, and about 26% at ten layers, as reported in Old can be Gold.

The strongest existing spiking normalization on our question is TAB (TAB) adapted to graphs. TAB uses causal statistics accumulated over time with weights $\omega[t]$ for each step. In full-graph training it centers each feature across nodes at every step, so it is the closest competitor and the one most likely to match pinning. We also run tdBN (Going Deeper), with $\alpha$ tuned, and STFN (Exploiting Spiking Dynamics), each inserted at the same positions in the network.

From the graph side, we run PairNorm placed before the spiking neuron, and a stacked combination of PairNorm followed by TAB. The stacked combination is the obvious "just combine them" design, and it gets its own row. We also port the isometric initialization and gradient-guided rewiring of Old can be Gold to the spiking network. That method reports about 80% on Cora at 64 layers and 72.28% on ogbn-arxiv for continuous GCN, the strongest deep-graph numbers in our sources.

DRSGNN, at 82.50±0.51% on Cora, is the strongest shallow spiking reference. The most important ablation is a learnable threshold for each layer, step and channel, trained freely rather than pinned to the population mean. It isolates whether the specific choice of pinning to the common mode matters or whether any time-varying threshold would do.

Every method uses the same splits, depths, step counts, surrogate gradient, optimizer and an equal hyperparameter-search budget of 30 trials per method per depth, selected on validation accuracy. We report means over 10 seeds. Pinning beats a baseline only if two things hold: its accuracy at depth 16 or more exceeds that baseline's by more than two standard errors, and its best accuracy over all depths is at least as high. Flattening the decay curve without matching the shallow peak counts as partial success, consistent with what PairNorm found.

No reported result was found on Lacuna for any iterative spiking graph network deeper than three layers. Every deep spiking number in this study will therefore be new.

Why not just…?

Why not just decouple propagation from spiking, as DRSGNN does? Decoupling avoids stacking spiking layers, but not depth itself. PairNorm shows that decoupled SGC drops sharply after four propagation steps on Cora and falls to 42.3% when unlabeled nodes have missing features, versus 74.5% with PairNorm. This is the setting where long propagation is needed. We include the decoupled design as a reference and evaluate it in the missing-feature setting, where the claim that depth matters can actually be tested.

Why not just add skip connections or rewiring? This is a serious threat on clean citation graphs, given the 64-layer result of Old can be Gold. However, skip connections carry continuous states around a threshold without controlling drift, and Going Deeper found that spiking residual networks failed to train without threshold-aware normalization. The ported rewiring baseline settles this directly. If it matches pinning, we will report that the bottleneck in spiking graph networks is gradient flow and not drift.

Why not just stack PairNorm and TAB? This is the most plausible competitor, and it may work. Both components center across nodes, so the stack already contains much of the pinning operation. The difference is twofold. Pinning acts on the membrane potential after carried-over charge is added, and it performs no rescaling, which removes the conflict between PairNorm's fixed total distance and TAB's per-feature variance. If the stack matches pinning at every depth, the contribution narrows to the diagnosis and to the cheaper inference form, which needs no computation across nodes. We will say so plainly.

Why not just use a high-pass filter channel? Complete the Missing Half gains roughly 10% on Texas and Cornell with a parallel high-pass channel. That channel adds continuous subtractions, which weakens the sparse-addition advantage of spiking hardware, and it does not address drift over time. We run a spiking filterbank on the heterophilic datasets. We note honestly that subtracting the neighbor mean is a local relative of pinning, so similar results would be informative rather than embarrassing.

Why not just let thresholds be learned, as DRSGNN does? The freely learned threshold ablation answers this. If thresholds trained without the pinning constraint reach the same accuracy at depth, the mean-tracking constraint is unnecessary. Our prediction is that they will not: thresholds fixed after training cannot follow how the common mode drifts as it depends on each input graph's features.

Proposed Experiments and Decisive Checks

The datasets are Cora, Citeseer and Pubmed under standard splits; Cora and Citeseer in PairNorm's missing-feature setting; ogbn-arxiv, to test scale; and Texas and Cornell, to test heterophily. The metrics are test accuracy, effective and numerical rank of membrane potentials and of spike matrices, $\rho(l,t)$, firing rate, and synaptic operation counts as an energy proxy, following STFN.

Four checks decide the outcome. First, across all unpinned runs, the collapse layer should correlate more strongly with $\rho$ than with the effective rank of membrane potentials, and the rank of spike matrices should collapse before the rank of membrane potentials. If $\rho$ predicts no better, the drift hypothesis is wrong. Second, slowing the leak should move collapse earlier only in unpinned networks. Third, pinning without any rescaling should hold accuracy at 32 to 64 layers, and pinning the input current instead of the membrane potential should not. If both versions work, Guo et al.'s mechanism is not the operative one here. Fourth, because of the warning in Are we measuring oversmoothing correctly?, we check whether pinning keeps rank high while accuracy still falls. We test this by adding feature noise and measuring whether the sign-detector behavior amplifies it. If so, the claim weakens from "prevents collapse" to "prevents uniform collapse but not loss of useful information," and $b_l$ or a dead zone around the threshold becomes the next design question. Finally, we report accuracy for the inference version that uses the frozen threshold table, since that version determines whether the method is usable on neuromorphic hardware.