In traditional artificial neural networks, optimization is almost entirely a quest for the right connection strengths, or weights. A weight determines how much one neuron influences another. However, in time-varying environments—such as audio processing, gesture recognition, or robotics—when an input arrives can be just as critical as its magnitude.
When processing temporal data, standard feedforward networks struggle because they lack a mechanism to retain information across time steps. To solve this, machine learning has traditionally turned to recurrent architectures (like RNNs or LSTMs) that pass a hidden state forward through time. But recurrence introduces complex feedback loops, high memory overhead, and training difficulties like exploding or vanishing gradients.
A powerful alternative approach optimizes the physical travel time of signals between neurons. By treating axonal and synaptic propagation delays as learnable parameters alongside weights, a simple feedforward Spiking Neural Network (SNN) can natively align asynchronous signals across time, effectively replacing recurrent loops with a lightweight, parameter-efficient form of short-term memory.
The Intuition: Coincidence Detection via Temporal Alignment
To understand why learnable delays work, consider how a biological or spiking neuron processes information. A spiking neuron acts as a leaky integrator: it receives incoming pulses (spikes), which temporarily elevate its internal membrane potential. If the potential crosses a specific threshold, the neuron fires. Crucially, because the membrane potential decays over time, spikes must arrive in close temporal proximity—a phenomenon known as coincidence detection—to trigger a response.
Imagine an auditory network trying to recognize a spoken word where feature $A$ occurs at time $t = 0$ ms and feature $B$ occurs at $t = 10$ ms. In a standard feedforward network, by the time feature $B$ arrives, the impact of feature $A$ has already decayed. Adjusting the weights (the amplitudes of the signals) cannot fix this fundamental temporal mismatch.
Without Delay Learning:
Feature A (t=0) ----> [Neuron] (Decays by t=10)
Feature B (t=10) ----> [Neuron] (Fails to cross threshold alone)
With Delay Learning:
Feature A (t=0) --[Delay=10ms]--> (Arrives at t=10) \
----> [Neuron] (Coincidence! Threshold crossed)
Feature B (t=10) --[Delay=0ms]---> (Arrives at t=10) /
By introducing adjustable transmission delays, the network can dynamically slow down the propagation of feature $A$ by exactly 10 ms. As a result, both signals arrive at the downstream neuron simultaneously.
As shown by Takashi Matsubara et al. in their Conduction Delay Learning Model for Unsupervised and Supervised Classification of Spatio-Temporal Spike Patterns, optimizing these conduction delays alongside weights allows neurons to act as highly effective coincidence detectors for complex spatio-temporal patterns. Similarly, Jibin Wu et al. demonstrated in Delay learning based on temporal coding in Spiking Neural Networks that learning axonal delays allows a model to precisely align critical information while delaying or suppressing irrelevant inputs, significantly enhancing temporal precision and network sparsity. This concept traces its roots back to early foundational algorithms like SpikeProp, introduced by Sander Bohte et al. in SpikeProp: backpropagation for networks of spiking neurons, which utilized multiple synaptic terminals with varying delays to expand a network's capacity for spatio-temporal pattern recognition.
Replacing Recurrence with Feedforward Buffering
Because learnable delays can hold onto a signal and release it at a specific future time step, they function as a passive, structural form of short-term memory. This insight allows feedforward networks to match or exceed the performance of recurrent architectures on sequence tasks without using any recurrent connections.
In Axonal Delay as a Short-Term Memory for Feed Forward Deep Spiking Neural Networks, Pengfei Sun et al. introduced a Rectified Axonal Delay (RAD) module. They demonstrated that allowing the network to learn transmission delays creates a temporal buffering mechanism. Instead of relying on active feedback loops to remember the past, the network simply shifts historical spikes into shared future temporal windows.
This feedforward alternative offers several distinct advantages:
- No Recurrent Loops: Eliminating feedback connections removes the risk of vanishing or exploding gradients during backpropagation through time.
- Parameter Efficiency: Rather than scaling up hidden state dimensions and weight matrices to capture longer contexts, the network simply optimizes a single delay parameter per connection.
- Long-Range Correlations: In Co-learning synaptic delays, weights and adaptation in spiking neural networks, Lucas Deckers et al. showed that co-learning weights and propagation delays enables simple feedforward structures to capture long-range temporal correlations that typically require complex recurrent loops.
To make this training stable, Pengfei Sun et al. proposed a two-stage optimization process in Adaptive Axonal Delays in Feedforward Spiking Neural Networks for Accurate Spoken Word Recognition. The network first undergoes a pre-training phase to establish a baseline delay distribution, followed by an adaptive phase that refines the temporal constraints based on the complexity of the input data.
Mathematical Formulation: Delays as Differentiable Convolutions
Historically, a major roadblock to training delays was differentiability. A delay shifts a signal discretely in time, making it difficult to compute standard gradients for backpropagation.
A breakthrough in framing this problem came from treating delays through the lens of convolutional neural networks. In Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings, Ilyass Hammouamri et al. showed that a synaptic delay can be mathematically represented as a 1D temporal convolution where the kernel contains a single non-zero value. By utilizing Dilated Convolutions with Learnable Spacings (DCLS), the temporal offset of a signal becomes a continuous parameter that can be optimized using standard gradient descent.
This formulation aligns with the theoretical proof provided by Xueyuan She et al. in Sequence Approximation using Feedforward Spiking Neural Network for Spatiotemporal Learning: Theory and Optimization Methods. The authors proved mathematically that feedforward SNNs can achieve universal approximation of any spike-sequence-to-spike-sequence mapping, provided they incorporate skip-layer connections and heterogeneous signal pathways with varying delays.
Hardware Efficiency and Extreme Quantization
Beyond algorithmic elegance, learnable delays are highly attractive for edge computing and neuromorphic hardware because they trade expensive arithmetic operations for cheap temporal offsets.
In traditional deep learning, high representational capacity requires high-precision weights (e.g., FP32). However, Pengfei Sun et al. demonstrated in Exploiting heterogeneous delays for efficient computation in low-bit neural networks that optimizing the arrival times of signals allows networks to achieve state-of-the-art performance on temporal tasks using extremely low-precision (ternary) weights. Because the temporal dimension itself acts as a primary computational resource, the network can drastically compress its memory footprint without sacrificing accuracy.
When deployed on physical neuromorphic chips, this paradigm yields massive efficiency gains. In A Complete Pipeline for deploying SNNs with Synaptic Delays on Loihi 2, Balázs Mészáros et al. built a pipeline to train SNNs with learnable synaptic delays on GPUs and deploy them directly onto Intel’s Loihi 2 chip. By offloading temporal memory to the physical properties of the hardware, they achieved a 250x reduction in energy consumption and an 18x speedup compared to a modern edge GPU.
This hardware realization fulfills a long-standing promise in neuromorphic engineering. As early as 2016, Peter U. Diehl et al. demonstrated in Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware that programmable hardware delays could successfully represent the hidden state of a recurrent network, proving that complex temporal dynamics can be offloaded entirely to physical transmission constraints.
Key Foundational Work
SpikeProp: backpropagation for networks of spiking neurons (2000) — contributed 1 concept elements to this direction.
Self-Organizing Network Learning of Sub-Millisecond Temporal Coded Information (1998) — contributed 1 concept elements to this direction.
Active Noise Canceling Using Analog Neuro-Chip with On-Chip Learning Capability (1998) — contributed 1 concept elements to this direction.