Spending a Latency Budget: Learnable Axonal Delays for Streaming CTC in Feedforward Spiking Networks

Spiking neural networks are networks of neurons that communicate with sparse binary pulses rather than dense numbers, and they are attractive for always-on, low-power speech hardware. Recent work shows that letting such a network learn axonal delays (how long each pulse takes to travel to the next neuron) gives a purely feedforward spiking network enough short-term memory to beat recurrent spiking networks on spoken-word classification, often with ten to twenty times fewer parameters. All of that evidence comes from classifying isolated, pre-cut words. Real recognizers must transcribe continuous speech whose word or phone boundaries are unknown. The standard tool for that is connectionist temporal classification (CTC), a training loss that sums over every possible alignment between input frames and output labels and lets the network output a "blank" symbol whenever it is not ready to commit.

The two ideas fit together in a specific way. In a causal, streaming network, CTC lets the model emit a label later than the sound that caused it, which buys right-hand context at the price of latency. Learned delays decide how much past the network can see at any instant. Together, the delay lengths set an upper bound on how long the network can usefully wait before emitting a label. We propose to make that bound an explicit, hardware-native budget. We would train causal feedforward spiking encoders with CTC under a fixed total delay budget, then ask whether learned, heterogeneous delays turn that budget into accuracy better than fixed delays, dense temporal convolutions, or recurrence.

Core Claim

Take a causal feedforward spiking encoder with per-synapse learnable delays trained with CTC. Its emission lag, meaning how many frames after a phone the network fires the corresponding label, is bounded by the sum of its per-layer delay caps plus a few membrane time constants. Our first claim is that this bound can be imposed directly as a budget $B$. Writing $\theta_l$ for the maximum delay allowed in layer $l$ of an $L$-layer network, the constraint is

$$\sum_{l=1}^{L} \theta_l \le B .$$

Our second claim is that, at every budget $B$, learning where to place delays inside this envelope gives a better phone-error-versus-lag curve than three alternatives at matched parameters: spreading fixed delays uniformly, using a dense causal convolution with the same reach, or using recurrent spiking layers. If a parameter-matched dense causal convolution reaches the same curve, the claim fails, and learnable delays are merely a reparameterized sparse convolution with no special value for sequence transduction.

This differs from simply "running a delay network with CTC." The object of study is the budget-to-accuracy tradeoff and how emission timing behaves under CTC. That is exactly the open question about the "waiting strategy" noted below, and it is a quantity a streaming deployment actually cares about.

Where this sits in prior work

The delay literature supplies the method and the motivation. Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings represents each synaptic delay as the position of a Gaussian kernel. That kernel is wide early in training and sharpened to an integer delay by the end, and the method reaches 95.07% on Spiking Heidelberg Digits with 0.2M parameters. We reuse this parameterization for GPU training. Learnable axonal delay in spiking neural networks improves spoken word recognition learns per-neuron axonal delays and reports that delay networks "wait," firing only after evidence accumulates. That paper classifies whole clips, so it never asks what waiting does to an alignment-based loss. That unanswered question is our motivation. Adaptive Axonal Delays in Feedforward Spiking Neural Networks for Accurate Spoken Word Recognition grows each layer's delay cap whenever too many neurons saturate at the cap. We adapt this scheduler so that it reallocates a fixed total budget across layers instead of growing caps freely. All three works evaluate only isolated words, and none constrains latency.

The CTC side supplies the task and constrains the design. Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition shows that recurrent leaky integrate-and-fire layers can serve as speech encoders trained with joint CTC and attention-decoder losses. They reach 4.69% word error rate on LibriSpeech 960h, against 3.46% for an LSTM encoder, and stay stable where plain recurrent networks diverge. They use recurrence and no delays, so they are our main spiking competitor. Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition uses CTC on a spiking transformer for handwriting, but its time axis is only two synthetic steps derived from images. It therefore says nothing about delays over physical time, although it warns that spike sparsity gives no wall-clock gain on GPUs. We take that warning as a reason to report synaptic-operation counts rather than GPU speed.

Breaking Through the Spike: Spike Window Decoding for Accelerated and Precise Automatic Speech Recognition shows that the frames right around CTC's sharp label peaks carry the useful context. That supports our use of peak timing as the measurable quantity, although its "spikes" are probability peaks from non-spiking models. Improving CTC-based Acoustic Model with Very Deep Residual Time-delay Neural Networks combines fixed-tap time-delay networks with CTC. It is the non-spiking ancestor of our dense-convolution baseline. Finally, A Complete Pipeline for deploying SNNs with Synaptic Delays on Loihi 2 requires integer delays, which is why we round to integers at the end of training and report accuracy after rounding.

Research Program

We would build a causal encoder of four to six feedforward leaky integrate-and-fire layers. Each synapse carries a learnable delay trained with the wide-to-sharp Gaussian scheme, and delays are rounded to integers in the last epochs. The output layer feeds membrane potentials into a softmax over phones plus blank, trained with plain CTC and no attention decoder, so that alignment behaviour is not confounded.

The budget is enforced in two ways, and we compare them. The first uses fixed per-layer caps summing to $B$. The second is a budgeted version of the adaptive scheduler from Adaptive Axonal Delays…. It moves cap from layers where few neurons saturate to layers where many do, while keeping the sum at $B$.

The analysis centres on emission lag. On a controlled testbed with known boundaries, and on TIMIT using its time-aligned phone segmentation, we measure the gap between each phone's end and the frame of its CTC peak. This answers the open tension about waiting directly. If CTC pushes the network to spend nearly the entire budget as lag, that shows waiting and CTC cooperate. If lag stays far below $B$ while accuracy still improves, the delays are being used as look-back memory rather than for waiting. A secondary analysis checks whether output-layer silence corresponds to blank, which would mean blank frames cost almost no output spikes.

Baselines

The naive reference point is the same feedforward spiking encoder with all delays set to zero, trained with CTC under identical data, epochs, and tuning budget. Following the 14–18% isolated-word gains from adding delays in Learnable axonal delay…, we expect a large gap. If the gap is small, continuous transcription does not need long memory in this regime.

The ablation that isolates our key ingredient is a fixed-delay network. It has the same architecture and the same per-layer caps, with delays spread uniformly or randomly in $[0, \theta_l]$ and frozen, while weights are learned. Learned delays must beat it at the same budget for "learning where to wait" to matter.

The strongest structural competitor is a dense causal spiking convolution whose kernel spans exactly $\theta_l$ frames per layer, as a spiking analogue of the residual time-delay CTC model. We run it in two forms: one with the same channel width, which has many more parameters, and one with channels reduced to match the delay network's parameter count.

The strongest spiking method is the recurrent leaky integrate-and-fire encoder of Bittar & Garner, which we would reimplement causally with CTC only, at matched parameters and the same hyperparameter-search budget. Its reported 4.69% LibriSpeech 960h word error rate used a joint CTC and attention-decoder setup, so it is not directly comparable to CTC-only phone recognition. No reported result was found on Lacuna for that model on TIMIT. We therefore count as beating it any equal or lower TIMIT phone error rate at equal or lower measured lag and parameter count.

As a non-causal upper reference we cite BLSTM-CTC, reported at 24.58% label error rate on TIMIT in Phoneme recognition in TIMIT with BLSTM-CTC. Approaching that figure with a causal spiking encoder at a modest budget would be a strong result. We do not require beating it, because that model sees the whole utterance.

All models share the same features (39-dimensional MFCCs, following the TIMIT CTC setup), the same 39-phone folding, greedy decoding, and the same number of tuning trials per model.

Why not just…?

Why not just delay the CTC targets by a fixed number of frames, or feed a fixed lookahead window? A global shift buys the same latency with no learned delays. This is exactly what the fixed-delay ablation and an explicit "shifted-input" variant test: at budget $B$, we give the zero-delay network $B$ frames of buffered input. If either variant matches learned delays, our gain is just latency, not structure. We would report that as a negative result.

Why not just use a dense temporal convolution? It sees every tap in its window, so it is strictly more expressive than one learned delay per synapse. The evidence that this falls short comes only from classification, where DCLS-Delays argues that sparse learned delays regularize better and Learnable axonal delay… reports seven times fewer parameters than CNNs on SHD. This is a real threat for continuous speech. The parameter-matched dense baseline settles it, and it is the comparison we treat as decisive.

Why not just keep recurrent spiking layers? Bittar & Garner shows they train stably at scale. Recurrence has no explicit latency knob, however, and on isolated words Adaptive Axonal Delays… reports 0.11 MB against 1.79 MB for recurrent spiking networks. We would measure the recurrent model's lag directly. If its error-versus-lag point lies on or above the delay curve, the feedforward design wins on controllability and memory. If it lies below, recurrence remains the better choice.

Proposed Experiments

We would first build a controlled testbed by concatenating Spiking Heidelberg Digits utterances with random silent gaps into digit strings. The task is CTC transcription, and exact boundaries are known, which makes emission lag measurable without error. This tests whether learned delays allocate the budget to cover inter-digit context and whether the budgeted scheduler moves cap toward deeper layers, as the unbudgeted scheduler did on isolated words.

The main experiment is causal TIMIT phone recognition. We sweep budgets of roughly 50, 100, 200, and 400 ms of total delay and plot phone error rate against measured median lag for every baseline. We also report parameter count, model memory, accuracy after integer rounding, and synaptic operations per second. The result is decisive if the learned-delay curve dominates both the fixed-delay ablation and the parameter-matched dense convolution at every budget, and reaches at least the recurrent baseline's error rate at no greater lag.

If TIMIT succeeds, we move to LibriSpeech 100h with CTC-only subword targets to test whether the advantage survives larger vocabularies and longer utterances. We would compare against the recurrent spiking encoder trained under the same CTC-only recipe, rather than against the published joint-decoder number. A failure on any decisive comparison would indicate whether delays matter only for isolated-word classification or whether their benefit was latency in disguise.