Is the Robustness Tax in Spiking Networks Really a Dead-Neuron Tax?

Spiking neural networks (SNNs) are brain-inspired models whose units communicate with binary pulses ("spikes") over a few discrete time steps rather than with continuous numbers. This makes them attractive for low-power chips, but deep SNNs have a known failure: activity fades as it passes through layers until later layers stop firing. Because training runs gradients through a smooth stand-in for the spike function, called a surrogate gradient, that is non-zero only near the firing threshold, silent neurons also stop learning. This is the "vanishing spike" or dead-neuron problem.

A separate literature asks why robustness training, especially adversarial training (training on worst-case perturbed inputs), lowers accuracy on clean test data. The two problems meet at one point. A robust spiking neuron should keep its internal voltage far from threshold, so that a small input change cannot flip its spike. A trainable spiking neuron needs its voltage near threshold, where the surrogate gradient is alive. We propose that in deep SNNs a measurable share of the clean-accuracy loss from robustness training comes from this conflict. Adversarial training pushes voltages out of the surrogate window, kills neurons in deep layers, and costs accuracy for reasons of trainability rather than representation. If that is right, the cost can be partly recovered by holding each layer's population of live neurons at its normally trained level while leaving the robust objective untouched.

Core Claim

The claim has a diagnostic part and an interventional part. The diagnostic part is that, for spiking residual networks trained with adversarial objectives, the clean-accuracy gap relative to normal training should grow with depth. It should also track a per-layer dead-neuron fraction and a drop in how much voltage mass sits inside the surrogate window. The interventional part is that a constraint keeping each layer's dead fraction near its normal-training reference should close part of the gap without lowering robust accuracy under strong, fixed attacks.

Both parts can be falsified. If the gap is flat in depth, or does not respond to restoring live neurons, then the SNN robustness tax is representational, the same trade-off seen in continuous networks. That negative result would still usefully redirect the field.

The intersection literature motivates this claim, though none of it tests it. SNN-RAT regularizes weights toward orthogonality to bound how much perturbations grow across spiking layers, and it notes that batch normalization hurts robustness. It never measures firing rates or dead units, and orthogonal weights cannot rescue a neuron whose input never reaches threshold. Robust Temporal Self-Ensemble (RTE) treats each time step as an ensemble member, making each robust and decorrelated from the others. It runs on SEW-ResNet backbones but does not examine whether its objectives change voltage distributions or silence neurons. RSC-SNN shows that Poisson input coding behaves like randomized smoothing and recovers clean accuracy by distilling from a continuous network. That recovery comes from an external teacher, not from repairing the student's own dynamics, and the paper stops at SEW-ResNet-18. The certified-bound work of Liang et al. identifies "unstable neurons" whose voltage interval straddles threshold as the obstacle to certification. This is the clearest statement in the file that robustness wants neurons away from threshold, and we turn that observation into a trainability question. We know of no prior work that measures dead-neuron dynamics under robust training or treats them as a mechanism of the clean-accuracy tax.

Research Program

The first component is instrumentation. For every spiking layer $l$ we record the dead fraction $D_l$, which is the share of neurons that emit no spike across all $T$ time steps on a clean input, averaged over the test set. We also record the surrogate occupancy $O_l$, the share of voltage values falling where the surrogate gradient exceeds a small fraction of its peak, and the mean firing rate. These are tracked through training for normal training and for each robust objective, across depths (SEW-ResNet-18, 34 and 50 in the style of Deep Residual Learning in Spiking Neural Networks) and across time windows $T \in \{2, 4, 6\}$. Short windows are included because temporal pruning work shows that low latency alone starves deep layers. The diagnostic prediction is that robust training raises $D_l$ and lowers $O_l$ more in deep layers than in shallow ones, and that the per-configuration clean gap regresses on the depth-summed increase in $D_l$.

The second component is the intervention, which we call firing-floor adversarial training. Let $D_l^{\text{ref}}$ be the dead fraction of the normally trained network at the same depth and $T$. We train the robust objective $\mathcal{L}_{\text{rob}}$ (PGD adversarial training or TRADES-style, following TRADES) subject to a per-layer ceiling on deadness, enforced with Lagrange multipliers:

$$\min_\theta \max_{\lambda \ge 0} \; \mathcal{L}_{\text{rob}}(\theta) + \sum_l \lambda_l \left( \tilde D_l(\theta) - D_l^{\text{ref}} - \delta \right)$$

Here $\tilde D_l$ is a differentiable relaxation of $D_l$: a sigmoid of the gap between threshold and each neuron's maximum voltage over time, computed on clean inputs. The tolerance $\delta$ is small, and $\lambda_l$ is updated by gradient ascent.

The constraint deliberately says nothing about how far above or below threshold a live neuron sits, so the margin that robustness needs stays free. This is the structural difference from shape penalties such as RecDis-SNN, which fixes a target voltage histogram for every layer. The multipliers also switch on only in layers that are actually dying, which keeps overhead low next to RecDis's reported 17% per-epoch cost. We will also test whether the constraint behaves differently in normalization-free networks initialized with SpikeInit. This matters because SNN-RAT's finding that batch normalization hurts robustness suggests the best robust backbone may be one that preserves firing without normalization. No one has tested SpikeInit under attack.

Attack evaluation needs care, because changing the training surrogate can hide gradients from the attacker. SNN-RAT showed that vanilla SNNs look robust until attacked through surrogate backpropagation through time, after which they drop to 0.04%. So every model is attacked with a fixed attack-side surrogate family that does not depend on the training surrogate. That family combines backpropagation through time (BPTT) and backpropagation through rate (BPTR), as in SNN-RAT, uses several surrogate widths, and adds a black-box transfer attack. Robust accuracy is the minimum over this ensemble.

Baselines

The naive reference point is a normally trained SEW-ResNet of the same depth and $T$, with no robustness objective. It fixes the clean ceiling and $D_l^{\text{ref}}$. The next reference is plain PGD adversarial training on the same backbone, which fixes the tax we are trying to explain. Every method shares the same data pipeline, epochs, attack budget ($\ell_\infty$, CIFAR-standard $\epsilon$), and hyperparameter search budget: an equal number of trials over learning rate and the method's own coefficient, selected on a held-out split by the average of clean and robust accuracy.

The strongest existing SNN competitor on this exact question is SNN-RAT, because it explicitly limits clean-accuracy loss. It reports a drop from about 93% to about 90% clean accuracy on CIFAR-10 with VGG-11, and 21.16% accuracy under strong PGD (SNN-RAT). We re-run it on our SEW-ResNet backbones under our attack ensemble rather than comparing to that figure, since its architecture and attack settings differ. RTE is the second strong competitor. It reports 59.50% clean accuracy on CIFAR-100 with competitive robustness (RTE), and we found no paired CIFAR-10 clean/robust figure for it on Lacuna, so it too is re-run. RSC-SNN's reported 58.94% PGD accuracy on CIFAR-10 with VGG-5 (RSC-SNN) uses input noise and distillation from a continuous network. We include it as a separate arm with and without its teacher, so that teacher-driven recovery is not confused with recovery from the network's own dynamics.

Our method beats a competitor if, at matched robust accuracy under the attack ensemble (within one standard deviation over three seeds), it recovers clean accuracy by an amount that grows with depth. It must also show a lower dead fraction. The ablation that isolates the key ingredient applies the same constraint to normal training, where there is little to recover, as a check that the constraint is not simply a general regularizer. A second ablation replaces the per-layer adaptive multipliers with one fixed global coefficient.

Why not just…?

Why not just widen or adapt the surrogate gradient? This is the most serious threat. A wider surrogate, or SpikeInit's adaptive surrogate that follows voltage shifts, would restore backward gradients to neurons far from threshold at almost no cost. Our argument against it is mechanistic. A wider surrogate gives gradient to neurons that still emit nothing in the forward pass, so deep layers still receive no information, and the surrogate's mismatch with the true step function grows. That argument could be wrong. We include width-swept and adaptive surrogates as full arms under the fixed attack ensemble. If they match firing-floor training on both clean accuracy and dead fraction, we will report the cheap fix as the answer, and the contribution becomes the depth-resolved accounting of the tax.

Why not just add RecDis's membrane-distribution loss to adversarial training? RecDis pushes voltages toward a two-peaked distribution, which roughly resembles a margin, so it could achieve both goals at once. It also penalizes voltages above 1 and fixes one histogram shape everywhere. That may fight the larger margins robust training wants in some layers, and it costs about 17% more time per epoch according to its authors. Adversarial training plus the RecDis loss is therefore a direct arm. Which approach is better is decided by the clean and robust numbers at matched training compute, not by argument.

Why not just use temporal batch normalization, which already stops vanishing spikes? SNN-RAT reports that batch normalization degrades SNN robustness, and SpikeInit shows it adds floating-point multiplies at inference that conflict with addition-only neuromorphic hardware. We still run a batch-normalized arm. If it matches our method's clean recovery without losing robust accuracy under the attack ensemble, then the dead-neuron mechanism is real but already solved in practice, and we would say so.

Why not just distill from a robust continuous network, as RSC-SNN does? Distillation recovers clean accuracy but needs a fully trained teacher and does not explain where the tax comes from. The teacher-on and teacher-off arms described above show whether distillation's gain overlaps with ours (the two recover the same accuracy) or adds to it. Evidence for overlap would be distillation lowering $D_l$.

Proposed Experiments and Decisive Checks

The main testbed is CIFAR-10 and CIFAR-100 with SEW-ResNet-18, 34 and 50 at $T \in \{2, 4, 6\}$, attacked with the BPTT/BPTR surrogate ensemble plus black-box transfer. Tiny-ImageNet is the scale check, since RTE and RSC-SNN both use it. For native event data we use CIFAR10-DVS (dataset) with perturbations that flip a limited number of events ($\ell_0$), as formalized by Liang et al. Here clean accuracy is anchored by RecDis-SNN's reported 72.42%. To check whether the mechanism extends past adversarial perturbations, we evaluate the same models on CIFAR-10-C-style common corruptions following the ImageNet-C protocol, using relative corruption error. We found no reported SNN result on Lacuna for CIFAR-10-C.

The first decisive check is the depth slope. We regress the clean gap between normal and adversarial training on depth and on the summed increase in dead fraction across nine configurations. A slope near zero falsifies the mechanism. The second check is the dose-response test: sweeping $\delta$ should trace a monotone curve in which lower dead fraction gives higher clean accuracy at unchanged robust accuracy. The third check guards against gradient masking. Robust accuracy under the fixed attack ensemble must not rise when the training surrogate changes, and if it does we treat that as possible gradient masking rather than a real gain. The fourth check tests normalization-free networks. SpikeInit-initialized networks under adversarial training should either show a smaller dead-neuron increase than batch-normalized ones, which would support settling the batch-normalization question in their favor, or a larger increase under noise, which is equally informative.

We also report per-layer firing rates and synaptic operation counts, because a method that restores accuracy by making every neuron fire more would erase the energy advantage that motivates SNNs in the first place.