Threshold-Flat Spiking Networks: Training for Accuracy That Survives Fewer Spikes

A spiking neural network sends binary pulses between neurons instead of continuous numbers. It saves energy because each pulse triggers only an addition, and a silent neuron costs almost nothing. Its dynamic energy therefore scales with how many spikes it fires. The obvious lever for cheaper inference is to make neurons fire less, usually by raising their firing thresholds. In practice this breaks accuracy, because a directly trained network settles where small threshold shifts flip many neurons on or off. We propose to train spiking networks to be flat along the threshold-raising direction. The idea borrows sharpness-aware minimization, a method that minimizes the worst-case loss in a small neighborhood around the current parameters. We apply its worst-case perturbation to the firing thresholds rather than the weights, and only in the direction that suppresses spikes. The resulting model should keep its accuracy as thresholds are raised at deployment. That gives a single knob for trading spikes, and hence energy, against accuracy without retraining.

The opportunity is that two literatures each hold half of the argument. Flat-minimum research shows that a model sitting in a broad, shallow basin of the loss tolerates parameter displacement. The second-order bound in Mirzadeh et al. ties loss increase to displacement size times curvature. Neuromorphic research shows that energy is set by spike count (Tavanaei et al.; Jang et al.). Raising thresholds is a parameter displacement whose effect on energy is direct and monotone. If the loss is flat along that displacement, energy can be cut at little accuracy cost.

Core Claim

Sharpness measured in weight space is the wrong quantity for an energy-constrained spiking network. The quantity that matters is curvature of the loss along the low-dimensional direction of increasing firing thresholds. It can be targeted cheaply because the threshold vector has one entry per layer or channel, not per weight. Concretely, let $w$ be the synaptic weights, $\theta$ the per-channel firing thresholds, $\mathcal{L}$ the task loss and $\rho$ a relative radius. We train

$$\min_{w,\theta}\ \max_{0 \le \delta \le \rho\,\theta} \mathcal{L}(w,\ \theta + \delta),$$

where the inner maximum is taken elementwise over non-negative threshold increases. It is approximated, as in standard sharpness-aware minimization, by one gradient-sign step clipped to the box. The constraint is one-sided because only increases reduce spikes. Negative perturbations would spend the flatness budget on a direction we never deploy along.

The prediction is concrete. At matched accuracy, a threshold-flat network should reach a lower synaptic-operation count after post-hoc threshold scaling than networks trained with plain surrogate gradients, weight-space sharpness-aware minimization, or explicit firing-rate penalties. Its accuracy-versus-spikes curve should degrade more gradually.

How This Differs From Existing Work

Three papers already connect landscape smoothness to spiking networks, and each supplies a piece we reuse or contrast against.

Adaptive Smoothing Gradient Learning learns the width of the surrogate gradient and mixes analog activations with "spike noise." This deforms an easy continuous landscape into the spiking one. It smooths the landscape the optimizer sees but never targets robustness to threshold changes or reports a spike-versus-accuracy curve under deployment-time adjustment.

Navigating Local Minima in Quantized Spiking Neural Networks escapes sharp minima with cosine learning-rate restarts. That is a heuristic with no explicit sharpness target and no energy measurement.

Robust Temporal Self-Ensemble shows that suppressing adversarial transfer across timesteps flattens the visualized loss surface. Its perturbations act on inputs and its goal is adversarial robustness, not energy.

None of the three perturbs thresholds, and none uses flatness as a route to fewer spikes.

RMP-Loss is the closest in spirit. It pushes membrane potentials away from ambiguous mid-range values toward 0 or the threshold. The research file raises an unresolved tension here. Clustering potentials near the threshold may make the loss sharper in parameter space, because a tiny shift flips those neurons. Our objective takes the opposite stance: it explicitly penalizes loss sensitivity to threshold motion. Measuring threshold-direction curvature under RMP-Loss is therefore a direct test of that tension.

On the flatness side, we reuse the one-step ascent of sharpness-aware minimization. We also reuse the observation from Make Continual Learning Stronger via C-Flat that curvature steps can be applied on a fraction of iterations to control cost. Overhead is a known open problem: C-Flat needs 2 forward and 4 backward passes per curvature step. That problem is sharper for spiking networks trained by backpropagation through time, which are already unrolled over $T$ timesteps.

Research Program

Optimizer. The first component is the optimizer itself. Thresholds are made learnable per channel, building on threshold-aware normalization from Going Deeper With Directly-Trained Larger Spiking Neural Networks (tdBN), which rescales inputs relative to the threshold. One full forward-backward pass through time gives gradients with respect to both weights and thresholds. We then take the clipped one-sided ascent step on thresholds only and compute the update gradient at the perturbed thresholds. To contain cost, the perturbed pass runs on a fixed fraction of iterations, swept from 10% to 100%, and training energy is reported alongside inference energy.

A complication flagged in the file is that perturbations can push membrane potentials across a hard step. That would make the surrogate gradient at the perturbed point unreliable. We monitor this directly by logging the fraction of neurons whose spike output flips under the ascent step. If the fraction is large, the one-step approximation is not tracking the true worst case.

Measurement. The second component is measurement. For every trained model we compute three quantities. The first is the largest Hessian eigenvalue of the loss with respect to weights, obtained by power iteration on surrogate-gradient Hessian-vector products. The second is the second derivative of the loss along the threshold-increase direction, estimated by finite differences on the true spiking forward pass, so it is not an artifact of the surrogate. The third is the full post-hoc curve of accuracy against synaptic operations as all thresholds are scaled by a common factor from 1.0 to 2.0.

The central analysis asks whether threshold-direction curvature, rather than weight-space curvature, predicts the area under that curve across all methods. If weight-space curvature predicts it equally well, the reframing is unnecessary.

Energy accounting. The third component is energy accounting. Inference energy is reported as accumulate operations, counted from spikes times fan-out, plus the multiply-accumulate operations of any non-spiking first layer. The comparisons that matter are ratios between methods, so we report counts directly and do not import hardware constants.

Where an Intel Loihi-class device is accessible, a subset of models is deployed to check that operation-count savings translate into measured energy. Prior deployments report large gains over conventional processors (Martí et al.; Tang et al.).

Baselines

Plain surrogate-gradient training (naive reference). This is a tdBN ResNet trained with backpropagation through time and a fixed surrogate (Zheng et al.). It reports 93.16% on CIFAR-10 and 96.87% on DVS128 Gesture. It shows what post-hoc threshold scaling does to an ordinary model. Every method shares its architecture, timestep count, data augmentation and epoch budget.

Adaptive Smoothing Gradient Learning (strongest existing smoothing method). Adaptive Smoothing Gradient Learning is the strongest spiking method in the file that explicitly reshapes the loss landscape toward minima tolerant of spike quantization. It reports 97.90% on DVS128 Gesture and 84.50% on CIFAR10-DVS. It also reports consuming 8.96% of the energy of its ANN counterpart on CIFAR-100, at a reported 94.11% accuracy that we will try to reproduce before relying on it.

RMP-Loss (membrane-margin regularizer). RMP-Loss reports 96.10% on CIFAR-10 with ResNet19 and 76.20% on CIFAR10-DVS. We include it because it embodies the opposite hypothesis about where potentials should sit relative to the threshold.

Explicit firing-rate penalty. This adds an L1 penalty on spike counts to the loss and sweeps its weight to trace a Pareto front. No reported result was found on Lacuna for this baseline on these benchmarks, so we run it ourselves. It is the simplest direct route to fewer spikes and the most dangerous competitor.

Weight-space sharpness-aware minimization. This is standard sharpness-aware minimization through the surrogate gradient, plus Lookbehind-SAM (Mordido et al.). In ANNs Lookbehind-SAM improves over ASAM on CIFAR-100 (82.16% vs 81.67%, WRN-28-10). No reported spiking-network result was found on Lacuna for either.

Ablations. Two ablations isolate the key ingredient. Random threshold jitter of the same magnitude tests whether the worst-case perturbation matters. A two-sided threshold perturbation tests whether one-sidedness matters.

Fairness and what counts as a win. All methods receive the same hyperparameter-search budget: 12 configurations each, including the radius $\rho$ for ours. Each method is reported with its own best post-hoc threshold scale, so no method is penalized for being evaluated at an unfavorable operating point. Beating the baselines means a strictly lower synaptic-operation count at accuracy within 0.5 points of the best baseline's unscaled accuracy, on at least three of the four benchmarks, over three seeds. The comparison is also made at matched total energy once training overhead is amortized over a stated number of inferences.

Why not just…?

Why not just penalize spike counts directly? This is the real threat. A firing-rate penalty is cheaper than our method and optimizes the energy target itself. Our argument is that it fixes one operating point per training run, and its gradient pushes potentials below threshold without regard to how many sit just under it. That could leave the network fragile to further threshold or hardware variation.

The experiment settles this in two ways. First, we compare full Pareto fronts: the penalty swept across retrainings against our model swept by post-hoc scaling. Second, we combine the two. If the penalty front dominates ours, the claim fails and we will report that the reframing adds nothing over the direct objective. If the combination beats both, flatness is complementary rather than redundant.

Why not just inject random threshold noise during training? Noise on inputs and weights is known to flatten landscapes (Lee). The randomized winner selection in RTK-WTA also improves robustness without any explicit sharpness term. Random jitter costs no extra backward pass.

The matched-magnitude jitter ablation is the decisive comparison. In a space with many threshold channels, random directions rarely align with the worst one, so we expect jitter to flatten the threshold direction less. If jitter closes most of the gap, we will switch to jitter and drop the claim that the worst-case step is the key ingredient.

Why not just use weight-space sharpness-aware minimization, or a stable training regime? Weight-space flatness might implicitly flatten the threshold direction, since thresholds and weight scale are partly interchangeable. A dropout-and-decay "stable regime" flattens landscapes for free in ANNs: Mirzadeh et al. report 59.9% on Split CIFAR-100 with forgetting of 0.08. Against the stable regime, Navigating Local Minima finds that dropout interacts badly with quantized spiking training, which already absorbs surrogate noise.

Against weight-space sharpness-aware minimization, the curvature analysis is the test. If weight-space eigenvalues predict the accuracy-versus-spikes curve as well as threshold-direction curvature does, and weight-space training matches our fronts, then the threshold framing is unnecessary.

Proposed Experiments

Benchmarks. Experiments run on CIFAR-10 and CIFAR-100 for static images, and on DVS128 Gesture and CIFAR10-DVS for event-camera data. These are the benchmarks where the file supplies baseline numbers. Models are ResNet19 and ResNet-34-scale tdBN networks at 4 to 6 timesteps. ImageNet with ResNet-34 serves as a scale check against the reported 67.05% (tdBN) and 65.17% (RMP-Loss) top-1.

Decisive checks. The first decisive check is the Pareto comparison described above. The second is the correlation analysis: across all trained models, threshold-direction curvature should explain variance in the area under the accuracy-versus-spikes curve that weight-space eigenvalues do not. The third resolves the RMP-Loss tension. If RMP-Loss models show high threshold-direction curvature and steep collapse under scaling, the file's suspicion is confirmed, namely that margin-pushing in voltage space sharpens the deployment-relevant landscape. If they show low curvature, our objective and RMP-Loss target the same property and should be merged.

Secondary check. A secondary check asks whether threshold-flat models are also more tolerant of 4-bit weight quantization, the regime studied in Navigating Local Minima, which reports 91.44% on DVS128 Gesture. That would show the flatness transfers to the other main source of on-chip error.

Cost reporting. All results report training accumulate- and multiply-operation overhead together with inference savings. This addresses the training-energy paradox the file identifies: a method that doubles training cost should repay it within a stated deployment horizon.