Spiking Gradients: Exact Integer All-Reduce via Integrate-and-Fire Compression

When many GPUs train one model, they must average their gradients at every step, and on slow links this exchange can dominate. Reported fractions run as high as 92% of training time in extreme large-language-model runs (Zhao et al.). Compressing gradients to one or two bits cuts the traffic but creates two practical problems. First, tiny integer codes overflow or lose accuracy when they are summed hop by hop around a ring, so systems fall back to slower gather-everything patterns. Second, the work of encoding competes with the backward pass for the GPU. Spiking neural networks face a closely related problem in a different field. Neuromorphic engineers convert ordinary networks into spiking ones, where a neuron accumulates input in a "membrane potential" and emits a discrete spike whenever that potential crosses a threshold. They have learned how to make a handful of discrete spikes over a few time-steps represent a continuous value with small, bounded error.

We propose to treat each worker's gradient stream as a bank of bipolar integrate-and-fire neurons whose "time-steps" are optimizer iterations. Each parameter integrates its gradient and emits a spike of −1, 0, or +1 (or a few integer levels). All workers share one threshold per parameter block, so the spikes can be summed exactly as small integers inside a standard int8 All-Reduce. The result needs no decompress–recompress at intermediate hops, no shared random seeds, no All-Gather fallback, and only constant-cost elementwise encoding. The membrane potential is provably bounded by the threshold, so the leftover error it carries can itself be stored in 8 bits rather than as a full-precision residual.

Core Claim

The mechanism is honest error feedback in spiking clothing. An integrate-and-fire neuron with reset-by-subtraction keeps a running remainder, which is what error-feedback compressors do. We do not claim the remainder idea is new. The claim is narrower and testable. Three properties together, borrowed from ANN-to-SNN conversion, change what gradient compression can do at the systems level.

The first is a threshold shared across workers. Spikes from $P$ workers then sum to exact integers in $[-PK, PK]$, where $K$ is the largest spike magnitude. This fits a native int8 ring All-Reduce whenever $PK \le 127$, so multi-hop reduction introduces zero additional error.

The second is rounding-to-nearest emission with symmetric bipolar spikes. In spiking terms this is the half-step shift of QCFS combined with the negative spikes of dual-threshold neurons. It makes the transmitted cumulative sum track the true cumulative sum to within half a threshold.

The third is a threshold-adaptation rule that targets a low clipping rate, in the spirit of learnable clipping in conversion. The bound above only holds when the threshold covers the gradient range.

Concretely, for a parameter on worker $p$ at step $t$, with gradient $g_t^p$, membrane $v_t^p$, block threshold $\theta_t$, and maximum spike magnitude $K$:

$$ s_t^p = \mathrm{clip}\big(\mathrm{round}((v_{t-1}^p + g_t^p)/\theta_t),\,-K,\,K\big), \qquad v_t^p = v_{t-1}^p + g_t^p - \theta_t s_t^p , $$

and the applied update is $\theta_t \sum_p s_t^p / P$. If no clipping occurs, $|v_t^p| \le \theta_t/2$ at every step. Telescoping the membrane equation then gives the drift bound below, where $\Theta$ is the largest threshold used up to step $t$:

$$ \Big|\sum_{\tau \le t}\sum_p g_\tau^p - \sum_{\tau \le t}\theta_\tau \sum_p s_\tau^p\Big| \le \tfrac{P}{2}\max_{\tau\le t}\theta_\tau . $$

This is the gradient analogue of a spiking neuron's firing rate approximating its input with error shrinking as $1/T$ over a window of $T$ steps. The bounded membrane is what permits storing $v$ in int8 relative to $\theta$, which unbounded error-feedback residuals do not allow.

What Is Borrowed and What Is New

The spiking-conversion literature supplies the method. Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks introduced QCFS, a staircase activation with a half-step shift and half-threshold initial membrane, and showed that it centers conversion error at zero under uniform-input assumptions. We reuse the centering. We note plainly that, in a symmetric bipolar setting, it reduces to round-to-nearest, and its value here comes only from being integrated across iterations.

Efficient ANN-SNN Conversion with Error Compensation Learning added dual-threshold neurons that fire negative spikes to cancel overshoot, reaching 94.75% on CIFAR-10 at two time-steps. Our bipolar spikes are exactly this device. The hardware objection raised against it, that unipolar neuromorphic chips cannot emit −1, does not apply to GPU integers. Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks learned clipping thresholds to trade clipping error against latency. We replace its gradient-trained threshold with a lagged statistic, because a threshold agreed on before communication cannot depend on the current step's gradients.

The gradient-compression literature supplies both the motivation and the constraints. DynamiQ identified the "compressor's dilemma": in multi-hop All-Reduce, re-quantizing partial sums compounds error. DynamiQ fixes this with correlated stochastic rounding under shared seeds and a fused decompress-accumulate-recompress kernel. We remove the re-quantization altogether by making partial sums exact integers.

BitDP sends 1- or 2-bit linear-layer gradients with per-channel scales. Because the codes cannot be summed directly, it uses All-Gather with local decoding, and it discards the residual. We keep the residual in a bounded membrane and use a shared scale so that a reduction collective works. AGoQ faces the same overflow issue at FP8 and resolves it with All-to-All plus local reduction. SDP4Bit and DeMo rely on Hadamard or discrete-cosine transforms to tame outliers. DeMo also uses momentum subtraction as implicit error feedback. Our encoder uses no transforms, and outliers are handled by spreading their mass over several steps of spikes rather than by rotating them away.

The strongest constraint comes from On the Utility of Gradient Compression in Distributed Training Systems. That study found compression beat well-overlapped 16-bit synchronous SGD in only 6 of over 200 configurations, because encoding contends with the backward pass. Our proposal is built to survive that finding, through an elementwise encoder and no collective change, or to be refuted by it.

Research Program

We will build a PyTorch DDP communication hook that holds an int8 membrane buffer per parameter and runs one fused kernel. The kernel adds the gradient, rounds, clips, and subtracts, then hands an int8 spike tensor to NCCL's native int8 sum All-Reduce. It rescales by the shared block threshold on receipt.

Thresholds are per weight matrix, so they cost one scalar per tensor. They are computed from the previous step's reduced update norm and a running clipping counter, and they are broadcast as part of the previous collective, which adds no extra round trip. The adaptation rule raises the threshold when the fraction of clipped entries exceeds a target, by analogy with the clipping/latency trade-off in Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks. It lowers the threshold when spike density falls too far.

For worker counts beyond $127/K$, we will test a hierarchical scheme. It uses int8 sums inside a node and int16 across nodes, which mirrors SDP4Bit's two-level precision split. Following BitDP's finding that norms and biases are sensitive, those tensors stay in BF16 at first. Whether spiking can include them is an explicit question.

On the analysis side, we will check the drift bound empirically by logging cumulative transmitted versus true gradient sums. We will measure the clipping rate and the uniformity of membrane values across bins. QCFS's zero-mean argument assumes that uniformity, and the file flags its violation in real networks as an open problem. We will also study how spike level $K$ trades bits against convergence. We will extend the error-feedback convergence argument to Adam-style optimizers by spiking the gradient before the moment updates, and we will compare that against spiking the final update.

Baselines

The naive reference point is uncompressed BF16 data-parallel training with PyTorch DDP's standard bucketed overlap. This is the configuration Agarwal et al. found nearly impossible to beat at high bandwidth. It defines the target loss curve and the wall-clock floor, and every method runs on identical data order, model, learning-rate schedule, and hardware.

The strongest existing low-bit competitor is BitDP's 2-bit mode, which shares our bit budget and setting. Its reported result on OpenWebText for GPT-350M is a validation perplexity of 22.74, against 22.17 for the uncompressed baseline (BitDP). It also reports speedups of 5.15× on InfiniBand and 7.47× on Ethernet for GPT-13B. We will reimplement it with the same per-channel scaling and All-Gather pipeline. Beating it means a lower perplexity gap than 0.57 at equal or better wall-clock time, with the advantage widening as worker count grows, since All-Gather traffic scales with $P$ and ours does not.

The strongest multi-hop competitor is DynamiQ. It reports reaching target accuracy up to 34.2% faster than MXFP8 and 40.8% faster than BF16 while retaining 99.9% of BF16 accuracy (DynamiQ). We will compare on its Wikitext-103 and TinyBERT-GLUE settings, with equal tuning budgets given to its bit-allocation knobs and to our threshold target. Beating it means equal accuracy retention at lower time-to-target, attributable to removing the decompress–recompress hop.

We also include FP8 gradient communication as in AGoQ's QuanGrad (AGoQ), and DeMo on its Dolma OLMo-300M setting. DeMo reports transmitting 7.5 MB versus 637 MB per step with no accuracy loss (DeMo), and we will compare bytes per step and time-to-loss.

Three ablations isolate the key ingredient. The first keeps our error feedback but uses per-worker scales with All-Gather, which isolates the shared-scale integer collective. The second removes the membrane and keeps the shared scale, which isolates error feedback. The third replaces round-to-nearest with stochastic rounding, which tests whether deterministic centering matters at all.

Why not just…?

Why not just keep BF16 and rely on overlap? At datacenter bandwidth this may win. Agarwal et al. showed that syncSGD beat compression in 194 of 200 configurations (On the Utility of Gradient Compression). We treat this as a real threat. We will run on full InfiniBand as well as throttled Ethernet, and we will report the encode kernel's time as a fraction of the backward pass. If spiking loses at high bandwidth, the claim narrows to the bandwidth-starved regime, where BitDP and Zhao et al. document large communication shares.

Why not just use standard error feedback with 1-bit or ternary codes? Mathematically, our encoder is error feedback, and we say so. The difference is what it enables: an integer-summable code under a shared scale, and a bounded residual that fits int8. Ablation one, with per-worker scales and All-Gather, is the exact experiment that tells whether these systems properties matter. If it matches us in wall-clock time, the spiking framing adds nothing and we will report that.

Why not just use FP8 or DynamiQ? FP8 gives only 2× compression over BF16, against 8× for int8 spikes with $K\le 1$ packed to 2 bits on the wire. DynamiQ already reports that MXFP8 drops accuracy by up to 2.5% in multi-hop reduction. DynamiQ itself is the serious competitor. Its correlated rounding needs a fused re-quantization kernel at every hop, which our exact integer sums remove. The head-to-head on DynamiQ's own benchmarks settles whether that removal buys measurable time.

Proposed Experiments

The main experiment pre-trains GPT-350M on OpenWebText on 8, 32, and 64 GPUs. It runs under full InfiniBand and under 10 Gbps Ethernet throttling, and we measure validation perplexity and wall-clock time-to-target-loss against all baselines. A secondary run uses OLMo-300M on Dolma for bytes-per-step comparison with DeMo. A third runs Wikitext-103 and TinyBERT-GLUE for comparison with DynamiQ.

Three checks are decisive. The first is whether the perplexity gap stays below BitDP's 0.57 at $K=1$. The second is whether time-to-target improves over BitDP as $P$ grows from 8 to 64, as the constant-size collective predicts. The third is whether the logged cumulative drift respects the $P\theta/2$ bound outside clipped entries. A negative result on the third, meaning frequent clipping on outlier-heavy layers, would show that lagged thresholds cannot replace Hadamard smoothing, and that is worth knowing.