Momentum Restarts to Settle Quantized Activations Before Spiking Conversion

A spiking neural network (SNN) is a network whose neurons communicate with binary pulses over a few discrete time steps. It can run on very little energy on neuromorphic chips. The most reliable way to build an accurate one is ANN-to-SNN conversion: train an ordinary network whose activations are already cut into a small number of discrete steps, then copy its weights into spiking neurons. With $T$ time steps, a neuron's firing rate can take only $T+1$ values. At very short windows ($T \le 4$), the converted network loses accuracy because some activations land on the wrong step. This proposal argues that part of that loss comes from how the source network was trained, and is not fixed by the conversion rules. During training with a straight-through estimator (a stand-in gradient that treats the staircase activation as if it were smooth), momentum keeps pushing pre-activations back and forth across step edges right up to the end of training. The network ends with many activations sitting on a step boundary. A spiking neuron, which sees its inputs arrive in uneven order, then rounds those borderline values the wrong way.

The intervention is borrowed from convex optimization. A scheduled restart wipes the optimizer's momentum and continues from the current point. It is known to stop accelerated methods from overshooting sharp minima. We propose triggering such restarts, layer by layer, whenever a layer's quantized activations start flipping bins. We test whether this leaves fewer borderline activations, and so a smaller conversion gap at low $T$. The method uses only unipolar spikes and adds no inference-time readout phase.

Core Claim

The claim has a diagnostic part and an intervention part. The diagnostic part is a measurable quantity, the boundary mass of a trained source network:

$$B_\delta = \Pr_{x,\,\ell,\,i}\left[\, \mathrm{dist}\!\left(z_{\ell i}(x),\ \mathcal{E}_\ell\right) < \delta\,\theta_\ell / T \,\right]$$

Here $z_{\ell i}(x)$ is the pre-activation of unit $i$ in layer $\ell$ on input $x$, $\mathcal{E}_\ell$ is the set of step edges of that layer's quantized activation, $\theta_\ell$ is its learned threshold, and $\delta \in (0,1)$ sets how close to an edge counts as borderline. We predict that $B_\delta$ at the end of training predicts the ANN-to-SNN accuracy gap at $T \le 4$ across runs that have matched ANN accuracy. The edge at zero matters most. Activations just below it are the ones that produce the "Case 1" error (the network output is zero but the SNN fires anyway), which Reducing ANN-SNN Conversion Error through Residual Membrane Potential identifies as the most damaging kind of unevenness error.

The intervention part is that momentum restarts triggered by bin flipping lower $B_\delta$ more than the obvious knobs do (learning-rate decay, momentum annealing, oscillation penalties) at the same ANN accuracy and compute. The lower boundary mass should then carry through to SNN accuracy.

Where This Sits

Three lines of prior work supply the pieces.

The first supplies the restart mechanism. Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent resets Nesterov momentum on a fixed epoch schedule so that noise accumulated in the momentum buffer is cleared. It reports lower error on deep continuous networks, but it does not study quantized activations and leaves the choice of schedule open. A Differential Equation for Modeling Nesterov's Accelerated Gradient Method explains why momentum overshoots: its damping vanishes over time, so it oscillates on sharp geometry. Sharpness, Restart and Acceleration shows that a logarithmic grid over restart lengths adapts to unknown sharpness at modest cost. We reuse that grid so the restart length does not need hand tuning. Restarting Algorithms: Sometimes There Is Free Lunch frames restarting as a black-box wrapper, which is what makes it attractive here: no change to the loss or the network.

The second line motivates the diagnosis and also constrains it. Overcoming Oscillations in Quantization-Aware Training shows that latent weights oscillate across quantization boundaries under straight-through training. These oscillations corrupt batch-norm statistics. The paper proves that scaling the gradient changes the amplitude of the oscillation but not its frequency, and it fixes the problem with a cosine-annealed penalty or by freezing weights. We move the object of study from weights to activations, because activations are what the spiking conversion discretizes. We also test whether removing momentum at the right moment, which is a cheaper intervention, achieves the same effect. Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets shows that the clipped-ReLU estimator gives gradients that correlate with the true gradient but are biased. Its analysis has no momentum, and that gap in the analysis is where we work.

The third line defines the target. The quantization clip-floor-shift activation (QCFS) in Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks centers the quantization error so that its expected value is zero. That guarantee assumes activations are uniform within each bin, and boundary mass measures exactly how that assumption fails. Trainable thresholds come from Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks and Efficient ANN-SNN Conversion with Error Compensation Learning. The latter also adds dual-threshold neurons that emit negative spikes, which are incompatible with unipolar chips. Masking based on residual membrane potential (SRP) in Reducing ANN-SNN Conversion Error through Residual Membrane Potential fixes Case 1 errors at inference, at the price of an extra readout window $\tau$. All three train the source network with ordinary optimizers. None of them asks whether the optimizer leaves activations in a convertible state.

Research Program

We will build flip-triggered restarts on top of Nesterov SGD with an increasing momentum coefficient, applied to a QCFS source network. The procedure has three steps. First, every $k$ iterations we compute, for each layer, the fraction of activations on a fixed probe batch that changed quantization bin since the previous check. Second, we track an exponential moving average of that flip rate. Third, when a layer's flip rate rises above a multiple $\rho$ of its running minimum, we zero the momentum for that layer's weights and threshold and restart its momentum coefficient.

We will compare this against a purely scheduled variant that uses the restart grid of Sharpness, Restart and Acceleration, to learn whether the trigger matters or whether restarting alone does. Both variants change only the optimizer state, never the loss, so a QCFS checkpoint trained with them converts with the unmodified QCFS rules.

The analysis has two parts. The first is the correlation between $B_\delta$, the Case 1 rate, and the gap at $T \in \{1,2,4,8\}$. We will measure the Case 1 rate using the negative-residual-potential criterion of Reducing ANN-SNN Conversion Error through Residual Membrane Potential, as a measurement only and not as a mask. The second part is a simple account of why restarts help. Near an edge, the straight-through gradient is roughly constant on both sides, so a momentum buffer built up on one side carries the pre-activation across. Zeroing the buffer leaves only a single gradient step, which is bounded by the learning rate times the gradient. That argument predicts that the benefit should grow as $T$ shrinks, since step edges get farther apart in absolute terms but the SNN becomes more sensitive to each misplaced activation.

Baselines

The naive reference is QCFS trained exactly as in Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks, using standard SGD with momentum and its step or cosine schedule. That paper reports 91.18% CIFAR-10 accuracy at $T=2$, and 69.37% on ImageNet with ResNet-34 at $T=32$. We will retrain it ourselves with the same seeds and epochs rather than rely on those reported figures.

The strongest existing unipolar, no-readout method trained from the same family is the learnable-clipping source network of Efficient ANN-SNN Conversion with Error Compensation Learning. It reports 94.75% on CIFAR-10 with ResNet-18 at $T=2$, though that figure includes dual-threshold neurons. We will therefore run it both with and without them, and treat the unipolar version as the fair comparison.

The strongest inference-side competitor is SRP, which reports 75.42% on CIFAR-100 with VGG-16 at $T=4$ and 64.32% on ImageNet with ResNet-34 at $T=10$, according to Reducing ANN-SNN Conversion Error through Residual Membrane Potential. We compare against it at matched total latency, $T$ plus $\tau$. We also test whether our method stacks with it.

The oscillation-side competitors are dampening and freezing from Overcoming Oscillations in Quantization-Aware Training, adapted from weights to activation pre-activations. That paper's weight-level results on 3-bit MobileNetV2 were 70.5% and 70.6%, against 64.97% for plain training and 69.50% for re-estimating batch norm after training.

Three ablations isolate the key ingredient. The first anneals momentum to zero over the final epochs. The second uses a fixed SRSGD restart schedule with no trigger. The third uses a random trigger with the same restart count.

Every method gets the same architecture, data, epochs, and a tuning budget of 12 configurations, searched on a held-out split. Each is reported over three seeds. Beating a baseline means a gain of at least 0.5 points in SNN accuracy at $T \le 4$ that falls outside the seed spread, at ANN accuracy within 0.2 points.

Why not just…?

Why not just anneal momentum or the learning rate at the end of training? This is the most serious threat. If borderline activations come only from the dynamics late in training, then decaying momentum to zero may do everything restarts do. We run momentum annealing as an explicit ablation with the same tuning budget. The restart hypothesis predicts a difference in two places. Restarts keep acceleration between resets, so they should reach the target ANN accuracy in fewer epochs, as Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent found for continuous networks. Restarts should also give lower $B_\delta$ at equal accuracy. If annealing matches on both, we will report that end-of-training momentum, not restarting as such, controls convertibility. That would still be a useful result, and the diagnostic would stand on its own.

Why not just apply oscillation dampening or freezing to activations? Overcoming Oscillations in Quantization-Aware Training reports that dampening is sensitive to the strength and timing of its penalty, and that freezing permanently removes capacity. An activation penalty also changes the loss, so it may push activations toward bin centers at a cost to ANN accuracy. The head-to-head comparison at matched ANN accuracy settles this question. If dampening wins, restarts lose.

Why not just fix errors at inference with SRP or dual-threshold neurons? Dual-threshold neurons need negative spikes, which standard unipolar chips do not support. SRP spends extra time steps $\tau$ on readout. Our method costs nothing at inference. The decisive comparison is accuracy at equal total time steps. We also test the stacked combination: if SRP plus restarts beats SRP alone, the two address different parts of the error.

Why not just re-estimate batch norm after training? The weight-level evidence in Overcoming Oscillations in Quantization-Aware Training shows that re-estimation recovers part of the loss but not all of it, falling short of active damping. We apply re-estimation to every method, so any remaining gain cannot be attributed to stale statistics.

Proposed Experiments

The main grid covers CIFAR-10 with ResNet-18 and VGG-16, CIFAR-100 with VGG-16, and ImageNet with ResNet-34. We evaluate SNN accuracy at $T \in \{1,2,4,8,16\}$, the ANN-to-SNN gap, $B_\delta$ for $\delta \in \{0.05, 0.1\}$, the Case 1 rate, the flip rate in the final epochs, and the number of epochs needed to reach target ANN accuracy.

The first decisive check is correlational. Across all runs, including deliberately varied momentum settings, $B_\delta$ should predict the gap at $T=2$ better than ANN accuracy does. If it does not, the mechanism is wrong.

The second check is interventional. Flip-triggered restarts must lower both $B_\delta$ and the gap compared with QCFS, momentum annealing, and the random-trigger ablation, at matched ANN accuracy.

The third check is the predicted scaling. The benefit should shrink as $T$ grows, and it should be largest on the zero edge, as measured by the Case 1 rate.

The fourth check covers the extreme regime. At $T=1$, where Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks reports collapse, we test whether fewer borderline units recover any accuracy, or whether capacity alone limits accuracy there. That would bound what training can ever fix.