Spike Flips and the Stability of Decentralized Spiking Network Training

Spiking neural networks (SNNs) are networks whose neurons emit all-or-nothing pulses ("spikes") when an internal voltage crosses a threshold. They are attractive for battery-powered devices that learn together without a central server, and one study reports that, in federated training on skewed client data, they lose far less accuracy than ordinary networks as the number of clients grows (Federated Learning With Spiking Neural Networks). Nobody can say why. The standard tool for explaining generalization from training dynamics is algorithmic stability, which asks how much the trained weights move when one training example is swapped. Every stability proof we know of assumes the weight update is a smooth function of the weights. Spiking training breaks that assumption twice. The backward pass uses a made-up "surrogate" derivative in place of the true derivative of the threshold, which is zero almost everywhere. The forward pass is also discontinuous, because a tiny weight change can flip a neuron from firing to silent and change the whole update.

This note proposes that the second break, not the first, is what matters. Take two copies of a decentralized run that differ in a single training example. Their weights drift apart for two reasons: smooth expansion through the surrogate derivative, and discrete jumps whenever the two copies disagree on whether a neuron spikes. The jump rate is governed by a measurable quantity, the density of membrane voltages sitting just below or above threshold. The result is a stability bound for decentralized surrogate-gradient training that has a concrete, controllable knob. It gives a falsifiable account of when spiking networks should resist client drift and when they should not, and it suggests a cheap intervention: push voltages away from the threshold.

Core Claim

Consider decentralized SGD on $m$ nodes, each holding $n$ samples, with gossip averaging over a mixing matrix. Couple two runs that differ in one sample at one node, and let $\Delta_t$ be the node-averaged parameter distance between the coupled runs at step $t$. Our central claim is a growth recursion of the form

$$ \mathbb{E}\,\Delta_{t+1} \;\le\; \bigl(1 + \eta_t(\beta_\sigma + G L_U \rho_t)\bigr)\,\mathbb{E}\,\Delta_t \;+\; \frac{2\eta_t G}{n}, $$

where $\eta_t$ is the step size and the remaining symbols are as follows. $\beta_\sigma$ is the Lipschitz constant of the surrogate update with the spike pattern held fixed; it scales with the peak height of the surrogate derivative, so narrow surrogates make it large. $G$ bounds the surrogate gradient norm. $L_U$ bounds how fast membrane voltages move with the weights. $\rho_t$ is the probability density of pre-threshold voltages at the threshold, averaged over neurons, time steps, and data at step $t$.

The term $G L_U \rho_t$ is the new piece. A weight perturbation of size $\Delta$ shifts voltages by about $L_U\Delta$, which flips a fraction of spikes of order $\rho_t L_U \Delta$. Each flip changes the update by at most $G$. Gossip averaging is non-expansive, so it enters, as in Stability and Generalization Analysis of Decentralized SGD and Unveiling the Power of Multiple Gossip Steps, through how fast the swapped node's divergence spreads, and that speed depends on the spectral gap of the communication graph.

Data heterogeneity enters through $\rho_t$. Our hypothesis is that skewed local data inflates the near-threshold voltage mass that a node's model produces after gossip brings in neighbors' weights. That would tie the consensus distance of Consensus Control for Decentralized Deep Learning to the jump term.

The bound makes three predictions that can be checked. First, surrogate width trades bias against stability: wider surrogates lower $\beta_\sigma$ but make the update less faithful to the forward computation. Second, the measured coupled-run divergence should track $\rho_t$ beyond what the surrogate's smoothness alone explains. Third, a regularizer that penalizes voltage mass near threshold should shrink the decentralized generalization gap. If the spiking advantage reported by FedSNN is real, it should coincide with a smaller effective expansion $\beta_\sigma + G L_U \rho_t$ than a matched continuous network.

Relation to Prior Work

Three stability analyses of decentralized training supply our proof machinery, and all three assume smooth or Hölder-smooth true gradients:

We reuse their decomposition of gossip and on-average stability. We replace the smooth gradient with a piecewise-smooth surrogate field whose pieces are indexed by spike patterns. Understanding Generalization of Federated Learning via Stability shows that heterogeneity enters stability linearly in a distribution distance. It does so for server-based aggregation with smooth losses, and we test whether heterogeneity in spiking networks acts instead through $\rho_t$. The original stability analysis of SGD in Train faster, generalize better provides the expansiveness-recursion template that our equation extends.

Two spiking-network papers study a different kind of stability and constrain our design:

Both concern how outputs respond to perturbed inputs, not how weights respond to a swapped training sample. We borrow the first paper's lesson by stating everything in terms of voltages, and we treat the second as a hint that $\rho_t$ may be small at initialization.

On the empirical side, FedSNN motivates the work: it reports the spiking advantage and explicitly calls for a theory. We note that it used server-based FedAvg, not peer-to-peer gossip, so the effect has not been observed in our setting. AS-FedBridge handles federations that mix spiking and continuous clients through distillation and has no stability content. Spatio-Temporal Backpropagation supplies the surrogate-gradient training method we analyze.

Research Program

The theoretical component proves the recursion under explicit assumptions. We assume a bounded surrogate gradient. We assume Lipschitz voltages in the weights over a finite unroll horizon, which follows from the decay factor below one and bounded inputs. We assume a bounded density of voltages at threshold. The proof first covers convex readout layers on fixed spiking features, then the non-convex case in the style of the Hardt et al. non-convex result, and it tracks the spectral-gap dependence through gossip. The key lemma converts expected spike disagreement between coupled runs into $\rho_t L_U \Delta_t$. The reset term complicates this, because a flip at one time step changes later voltages. We handle it by bounding the cascade over the finite horizon, and we report how the constant grows with the number of time steps rather than hiding it.

The measurement component makes the bound observable. We run coupled decentralized trainings that share all randomness and differ in one sample at one node. At every step we log three quantities: parameter divergence, the empirical spike-flip rate between the coupled runs, and a kernel estimate of $\rho_t$. This lets us split the observed divergence into a part explained by surrogate smoothness and a part explained by flips.

The intervention component adds a margin penalty on near-threshold voltage mass. Its single hyperparameter is chosen on validation data, and we test whether it lowers $\rho_t$, coupled-run divergence, and the train-test gap together.

Baselines

The naive reference point is plain decentralized averaging (DFedAvg/D-SGD) with a continuous ReLU network on the same ring and Dirichlet partitions. On CIFAR-10 with Dirichlet $\alpha=0.3$, Improving the Model Consistency of Decentralized Federated Learning reports 73.55% validation accuracy for DFedAvg and 82.39% for server-based FedAvg.

The strongest existing gap-closing method is DFedSAM-MGS from the same paper. It runs sharpness-aware local steps, which push each node toward flat regions of the loss, and adds several gossip rounds per update. It reports 84.26% on that setting. It is the strongest competitor because it attacks exactly the inconsistency we claim to explain, and it can be applied to spiking networks as well. We will run it on both continuous and spiking models with the same number of local steps, the same total communication budget, and the same tuning grid as our margin regularizer. Our method beats it if the margin regularizer, at equal or lower communication, matches its accuracy on spiking networks. We also count it as a win if adding the regularizer on top of DFedSAM-MGS improves accuracy, because that would show the two act on different things.

No reported result was found on Lacuna for any decentralized gossip-based spiking network on CIFAR-10, CIFAR-100, or CIFAR10-DVS. We will therefore establish the spiking DFedAvg number ourselves. We also include FedGuCci+ (Improving Group Connectivity), which keeps nodes near historical global models and reports 35.34% versus 25.60% for FedAvg under extreme heterogeneity on CIFAR-100.

The ablation that isolates our key ingredient is a smoothed continuous twin. This network replaces the hard threshold in the forward pass with the same sigmoid whose derivative is the surrogate. It shares the same weights, normalization, and time unrolling. The twin has $\beta_\sigma$ but, by construction, no jump term. A second ablation sweeps surrogate width with $\rho$ held roughly fixed.

Why not just…?

Why not just treat the surrogate as a biased gradient and use existing inexact-gradient stability arguments? This is the most serious threat. If the surrogate field were a smooth function of the weights, a bias term would suffice and our jump term would be unnecessary. The surrogate field is not smooth in the weights, though, because it is evaluated at a spike pattern that changes discontinuously. The decisive experiment is the divergence decomposition. If the flip rate explains a negligible share of coupled-run divergence, and the smoothed twin diverges exactly like the spiking network, then the simpler bias view wins and we will say so. We would then fall back to a weaker contribution: the first decentralized stability bound for surrogate training.

Why not just compare against a better continuous baseline, since the FedSNN gap may be an artifact? We think this is plausible. FedSNN's spiking models used Batch Normalization Through Time, a per-time-step normalization, and its continuous baseline may not have been matched. The smoothed twin and a continuous network with identical normalization are therefore mandatory baselines, not extras. If the spiking advantage disappears under matched baselines in gossip training, the explanatory goal shifts. The question becomes whether $\rho_t$ predicts the gap across architectures and partitions, which the theory still answers.

Why not just widen the network or add gossip steps? Widening the Network Mitigates the Impact of Data Heterogeneity on FedAvg reports that widening cut the CIFAR-10 non-IID penalty from 44.6% to 14.8%. That leaves a real deficit and costs memory that edge devices lack. Unveiling the Power of Multiple Gossip Steps reports CIFAR-100 accuracy falling from 30.50% to 29.66% when gossip steps rise from 3 to 10 at batch size 96. We include width and gossip-step sweeps so that we can check whether the margin regularizer is complementary to them. Our bound also predicts that widening a spiking network changes $\rho_t$, and we will check that prediction.

Proposed Experiments

Training uses spatio-temporal backpropagation with leaky integrate-and-fire neurons on four datasets: CIFAR-10 and CIFAR-100 with Dirichlet $\alpha \in \{0.1, 0.3, 1.0\}$, and CIFAR10-DVS and N-MNIST for event-stream data. The centralized STBP reference on N-MNIST is 98.78% Spatio-Temporal Backpropagation. We use 16 to 64 nodes on ring, torus, and fully connected graphs, so the spectral gap varies. The metrics are test accuracy, the train-test loss gap, consensus distance, coupled-run divergence, spike-flip rate, and $\rho_t$.

The first decisive check asks whether the measured $\rho_t$, combined with $\beta_\sigma$, predicts the rank order of generalization gaps across surrogate widths, topologies, and heterogeneity levels better than consensus distance alone. We measure this with rank correlation over all runs.

The second decisive check is a dose-response test. Increasing the margin penalty should lower $\rho_t$, coupled-run divergence, and the gap together. If accuracy improves without $\rho_t$ falling, the proposed mechanism is wrong.

The third check revisits the reported spiking advantage under gossip and asks whether it survives the matched continuous twin. If it does, the check asks whether the advantage's size tracks the difference in effective expansion between the two models. Each check can fail on its own, and the theory is useful only if the first two pass.