Retimed Spiking DAGs: Training Irregular Spiking Networks Under the Timing They Will Actually Run With

Neural networks are usually built as stacks of layers, each finished before the next begins. A different line of work treats a network as an arbitrary wiring diagram, a directed acyclic graph (DAG) of individual neurons such as random, small-world or scale-free graphs. It makes training tractable by regrouping neurons into "virtual layers": sets of neurons that do not depend on each other and can therefore be computed together in one matrix operation. Spiking neural networks (SNNs) are networks whose neurons communicate with binary pulses over discrete timesteps. They save energy because a pulse times a weight is just the weight, so each incoming spike costs an addition rather than a multiplication. Combining the two directions looks easy: put spiking neurons on the irregular graph and reuse the virtual layers. The difficulty is time. On neuromorphic chips a spike typically takes at least one clock tick to cross a synapse. A neuron that sits at the end of both a long path and a short path therefore receives information about different moments of the input. Training methods that evaluate the whole graph within a single timestep never see this mismatch.

This note proposes to treat the virtual-layer index of each neuron as its arrival time and to make edge delays an explicit design choice. A neuron's virtual-layer index is its longest-path distance from the input. When every edge takes one tick, all neurons can be updated in parallel at every tick, and a spike from input neuron $u$ first reaches neuron $v$ exactly $\mathrm{level}(v)-\mathrm{level}(u)$ ticks later along the longest path. Shorter paths deliver it earlier. Circuit designers handle this kind of mismatch with retiming: they add buffer delays to short edges so that all signals meeting at a node come from the same input moment. The proposal is to build spiking DAG networks whose training-time timing matches deployment timing, to use retiming to decide which edges get padded, and to test one central prediction. Full synchronization should help on inputs where only the current frame matters. Deliberately unsynchronized, multi-delay graphs should help on tasks with real temporal structure, such as spoken digits. Both should be compared against layered SNNs and zero-delay spiking DAGs at a matched synapse budget and a matched count of synaptic additions.

Core Claim

The closest existing work, 4Ward and its companion Breaking the Structure of Multilayer Perceptrons with Complex Topologies, does three things. It groups the nodes of an arbitrary DAG into levels by longest path. It trains them with a fixed binary mask on dense weights, $W = M \odot \tilde W$. It shows that such graphs beat multilayer perceptrons on hard Swiss Roll and S-Curve tasks. Its reported gains are up to 2.8x over a sequential baseline and up to 160x over deepstruct, with runtime scaling with graph height rather than node count. That work uses memoryless continuous neurons and treats levels as a scheduling convenience with no temporal meaning.

We claim that for stateful spiking neurons the levels are not just a scheduling device: they fix the timing of the computation. Three consequences follow, and the third is the testable core.

First, under unit-delay semantics the within-timestep level loop disappears. Every tick is one masked matrix product over all $N$ neurons, followed by a leaky integrate-and-fire update. A leaky integrate-and-fire (LIF) neuron accumulates input into a membrane potential that leaks over time and emits a spike when the potential crosses a threshold. Under this scheme the sequential work is $T+H$ ticks for $T$ input steps and graph height $H$, instead of $T \cdot H$ level evaluations under zero-delay semantics.

Second, the depth cost moves into output latency and into extra ticks during which neurons can fire. This makes the energy question a real empirical question rather than a free win.

Third, for each edge $e=(u\to v)$ with integer delay $d_e \ge 1$, define the input moment it carries relative to $v$'s arrival time. Retiming chooses the delays. Full retiming, $d_e = \mathrm{level}(v)-\mathrm{level}(u)$, makes every merge node integrate a single input moment, much as a layered network does. Setting $d_e=1$ everywhere gives each merge node a built-in temporal mixture whose time offsets are inherited from the graph's path-length distribution. The prediction is that the better choice depends on the task's temporal structure, and that small-world and scale-free graphs, with their broad path-length spread, benefit most from the unsynchronized setting on temporal tasks.

Two gaps in the research file partly dissolve under this formulation, and we should say so rather than claim them as contributions. The first is the "spike purity at merge nodes" worry. Spikingformer showed that unweighted addition of spike tensors in residual shortcuts creates values of 0, 1 and 2, which need costly integer-float multiplications. It fixed this by adding block outputs to the next neuron's membrane potential instead, reporting a 57.34% energy cut over Spikformer on ImageNet. In a neuron-level DAG every edge carries its own weight and feeds the membrane directly, so every merge is already a membrane-based shortcut. We forbid unweighted identity edges, which keeps spikes binary by construction. The second is the mask-trick waste. It is a GPU training cost, not a deployment cost. We will report energy as synaptic additions (spikes times actual out-degree), never as dense floating-point operations. This is the accounting that underlies the 41.7x figure of Direct training high-performance spiking neural networks for object recognition and detection and the roughly 1/274 figure of VTSNN.

Research Program

We will build a PyTorch spiking-DAG engine that reuses 4Ward's longest-path layering and mask trick as the training substrate. It adds per-edge integer delays, implemented as a ring buffer of the last $D_{\max}$ spike vectors, so that each tick computes

$$ V_v(t) = \beta V_v(t-1) + \sum_{d=1}^{D_{\max}} \big(M^{(d)} \odot \tilde W\big)_{:,v}^{\top} S(t-d) - \vartheta\, S_v(t-1), $$

where $V_v$ is neuron $v$'s membrane potential, $\beta$ the leak factor, $\vartheta$ the firing threshold, $S(t)$ the binary spike vector at tick $t$, and $M^{(d)}$ the mask of edges assigned delay $d$. Spikes fire when $V_v(t) > \vartheta$.

Training uses surrogate gradients with backpropagation through ticks. Normalization follows the threshold-dependent batch normalization of Going Deeper With Directly-Trained Larger Spiking Neural Networks, which keeps membrane variance tied to the threshold, applied per virtual layer. The loss is computed at every output tick as in TET, with the first $H$ latency ticks excluded under retimed and unit-delay semantics.

The program has three studies.

The first is a delay-semantics study. On a fixed graph family we compare three timing regimes. Zero-delay semantics is the direct 4Ward extension, in which each timestep evaluates all $H$ levels in order. Unit-delay semantics sets $d_e=1$ everywhere. Full retiming sets $d_e$ equal to the level difference.

The second is a partial-retiming study. Here we pad only edges whose level gap exceeds a threshold $k$, which traces a curve from unit-delay ($k=\infty$) to full retiming ($k=0$). This lets us find where accuracy changes and how the number of delay buffers grows.

The third is a deployment check. We map trained graphs with SNEAP, which partitions neurons into chip cores with the Metis graph-partitioning algorithm and then places cores on the chip's on-chip network using hop count as a proxy for communication energy. The question is whether the irregular graphs and the padded edges inflate routing cost relative to layered networks of the same size.

Baselines

The naive reference point is a layered spiking multilayer perceptron with the same neuron count and the same number of synapses. Its sparsity is introduced by random edge removal, and it is trained with the identical tdBN and TET recipe, tick budget and hyperparameter search (same number of trials per method). If the irregular graphs cannot beat this, the topology contributes nothing. A second layered baseline uses magnitude pruning to the same synapse count, starting from a dense layered SNN. This answers the reviewer who says pruning gives sparsity for free.

The strongest competitor on our specific question is the zero-delay spiking DAG: 4Ward's levels evaluated in order within each timestep, with the same graphs, neurons and training recipe. It is strongest because it gives every merge node perfectly current information at no latency cost, and it is the design anyone would build first. For speed comparisons we also give it the best available temporal parallelization. Each level depends only on earlier levels' full spike trains, so each level can be processed across all timesteps at once with the parallel formulation of Parallel Spiking Unit, cutting its sequential depth to $H$. Our method beats this baseline only if one of two things holds. Either unit-delay or retimed graphs match its accuracy within one standard deviation over five seeds while using fewer synaptic additions per correct inference. Or unsynchronized graphs exceed its accuracy on a temporal task.

The key-ingredient ablations are unit delay versus full retiming versus partial retiming on the same trained-from-scratch graphs. A further ablation shuffles delays: it keeps the delay histogram but randomizes which edges get which delay. This separates "having diverse delays" from "having delays aligned to path structure."

For external reference on the same benchmarks, the file reports these figures. On CIFAR10-DVS, TET reaches 83.17% top-1 (TET). On DVS128-Gesture, STBP-tdBN reaches 96.87% (Going Deeper With Directly-Trained Larger Spiking Neural Networks) and Spikingformer reaches 98.6% (Spikingformer). On Spiking Heidelberg Digits, a parallel recurrent PSU reaches 92.49% (Parallel Spiking Unit). On Swiss Roll and S-Curve no absolute accuracy was found on Lacuna; the file states only that DAG topologies significantly beat MLPs. Our neuron-level graphs of at most about $2^{12}$ neurons will not approach the convolutional and transformer numbers. These serve as ceilings, not targets, and the claims are made at a matched synapse budget.

Why not just…?

Why not just prune a standard layered SNN? Pruning gives sparsity, but it keeps every input-to-output path the same length. So it can never produce the path-length spread that our temporal-mixture hypothesis depends on. The only evidence in the file that irregular topology helps accuracy at all comes from continuous networks on synthetic manifolds (Breaking the Structure of Multilayer Perceptrons with Complex Topologies), so this is a genuine threat in the spiking case. The magnitude-pruned layered baseline at a matched synapse count settles it directly. If it wins everywhere, the proposal fails on topology and only the timing analysis survives.

Why not just run 4Ward's zero-delay levels and parallelize over time with PSU? On GPU training speed alone, this may well tie or beat unit-delay execution, since both reduce the sequential depth to about $H$. We will report that honestly rather than claim a speedup. The objection misses deployment, however. Zero-delay semantics assumes a whole chain of neurons settles within one timestep. Event-driven chips such as the ones behind the TrueNorth and Loihi results in the file advance spikes in ticks, so a zero-delay model is trained under timing it will not have. Our experiment trains in zero-delay mode, evaluates in unit-delay mode, and measures the accuracy drop. A small drop would make the objection win on practical grounds.

Why not just give every edge one tick of delay, as neural simulators already do? This is the unit-delay arm, and it costs nothing extra in memory. If it matches full retiming on every task, retiming is unnecessary and the contribution shrinks to the timing analysis plus the energy accounting. The partial-retiming sweep and the shuffled-delay control are designed to show whether alignment between delays and graph structure matters, on which tasks, and how much.

Why not just train any SNN and fix placement afterwards with SNEAP? SNEAP reports a 23% reduction in on-chip communication energy and a 51% reduction in spike latency through better neuron placement (SNEAP). That is complementary to this work, not a substitute. It neither chooses the network's timing nor trains irregular graphs, so we use it as the deployment step for every model, including the baselines.

Proposed Experiments

The graph families are Erdős–Rényi (random), Watts–Strogatz (small-world) and Barabási–Albert (scale-free) DAGs, with $N \in \{2^8, 2^{10}, 2^{12}\}$ neurons and connection densities matching the 4Ward scaling benchmark. Tasks span a timing spectrum. Swiss Roll and S-Curve with up to 12 segments are rate-encoded, a regime where only the current input matters and full retiming should win. DVS128-Gesture and CIFAR10-DVS, after spatial downsampling to fit neuron-level graphs, are intermediate cases. Spiking Heidelberg Digits is strongly temporal, so unsynchronized graphs should win there.

The metrics are accuracy over five seeds. Energy is measured as synaptic additions per inference and as spikes per inference. A chip-level estimate converts spike counts at 22 pJ per spike, using the transceiver figure of A 22-pJ/spike 73-Mspikes/s 130k-compartment neural array transceiver, and is stated to be specific to that chip. We also report time to first correct output in ticks, GPU wall-clock time per epoch, and average hop count after SNEAP mapping.

The decisive checks are three. The first is a sign flip in the ranking of retimed versus unit-delay graphs between the manifold tasks and Spiking Heidelberg Digits. Its absence would falsify the central hypothesis. The second is the accuracy lost when zero-delay-trained graphs are evaluated with unit delays. A large loss means the timing-faithful training proposed here is necessary, and a negligible loss means it is a curiosity. The third is whether any irregular graph, under any timing regime, beats the magnitude-pruned layered SNN on accuracy per synaptic addition. A yes would be the first evidence in the file that the topology advantage reported for continuous networks transfers to spiking networks once honest timing and energy accounting are imposed.