Spiking neural networks are neural nets whose units communicate with sparse binary pulses ("spikes") and carry a decaying internal voltage from one step to the next. Running a robot controller as a spiking network on neuromorphic chips can cut energy per decision by one to two orders of magnitude. The usual way to train such a controller with reinforcement learning is asymmetric. A spiking "actor" (the policy that outputs actions) is trained by an ordinary deep "critic" (a network that scores how good an action is). The critic is thrown away after training. Every published system of this kind assumes the robot senses and acts on a fixed clock. Real event-driven robots do not. Event cameras, asynchronous inertial sensors, and variable compute latency make the gap between successive decisions jitter. Under jitter, a spiking actor's voltages integrate over physically different amounts of time, so the same observation sequence produces different internal states and different actions.
We propose that the fix belongs mostly in the critic, the part that costs nothing at deployment. The critic should be a small causal Transformer that reads the recent window of observations and actions, each tagged with a continuous-time encoding of when it happened. It should also discount rewards by elapsed physical time instead of step count. The claim has two parts. First, under irregular timing a critic that sees only the current state and action, even with the time gap appended, scores the spiking actor's behavior wrongly. The actor's action depends on a hidden voltage history that the critic cannot see. Second, an attention critic with temporal positional encoding can reconstruct that history and train a better spiking actor without adding a single operation to the deployed chip.
Core Claim
The closest prior systems each cover one piece of this. Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks gives the critic a fixed-length buffer of past actions while the spiking actor runs without any history. This motivated our approach: it shows the critic needs history to cope with the actor's internal state, and it reached 0.04 m position error on a real quadrotor. Its buffer is a plain fixed-interval concatenation with no notion of elapsed time. Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns supplies the method. It makes the critic a two-layer causal Transformer over state-action chunks and averages the gradients of multi-step Bellman losses. Its actor is a standard deep network, however, and its positions are integer step indices. Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization constrains the design. It shows that discrete temporal-difference learning degrades as the control interval shrinks, and it fixes this for multilayer-perceptron actor-critics by scaling the learning rate with the interval. It never handles intervals that vary within an episode or a stateful actor.
What is new here is the pairing of a time-encoded sequence critic with a stateful spiking actor under jittered control intervals. We will also make a testable prediction about why the pairing matters. The benefit should grow with timing jitter, vanish at a fixed clock, and largely disappear when the spiking actor is replaced by a memoryless feedforward actor. If the gain does not depend on the actor having memory, our mechanistic story is wrong, even if the method still helps.
Research Program
The critic receives a window of the last $K$ decisions as tokens $(o_k, a_k, \Delta t_k)$. Here $o_k$ is the observation, $a_k$ the action, and $\Delta t_k$ the physical time since the previous decision. Each token's absolute offset from the window's end, $\tau_k$, is mapped through sinusoidal features at several frequencies, in the style of the continuous-time encoding used in Transformer Networks for Trajectory Forecasting. That work encodes trajectory timestamps this way for pedestrian forecasting; we reuse the encoding for value estimation. The pairwise interval between tokens is also added as a learned attention bias, following the relative-interval embeddings used in sequential recommendation (Exploring Periodicity and Interactivity in Multi-Interest Framework for Sequential Recommendation). Value targets discount by elapsed time rather than step count:
$$y_t = \sum_{k=0}^{n-1} \gamma^{\,t_{k}-t_0}\, r_k \;+\; \gamma^{\,t_n - t_0}\, Q_{\bar\phi}(h_{t_n}).$$
In this target, $t_k$ is the wall-clock time of the $k$-th decision in the chunk, $r_k$ is the reward accumulated over interval $k$, $\gamma$ is a per-second discount, and $Q_{\bar\phi}(h_{t_n})$ is the critic's value of the windowed history at the chunk's end. The critic loss averages gradients over the $n$ partial targets as in T-SAC. The learning-rate scaling from CSAC is applied using the mean interval.
The actor is a population-coded spiking network in the style of Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control (PopSAN). It is trained with the adaptive surrogate-gradient schedule and jump-start warm-up of the quadrotor work above. A surrogate gradient is a smooth stand-in for the spike function's zero-or-undefined derivative, used only in the backward pass. Its leaky integrate-and-fire neurons use the exact decay $e^{-\Delta t_k/\tau_m}$ for the elapsed interval, where $\tau_m$ is the membrane time constant. This costs one lookup per step and keeps spikes binary.
As a secondary question, we test whether a spike-native timing signal adds anything on top of exact decay. Here the firing thresholds are modulated by $\Delta t$, adapting the PE-LIF neurons of Positional Encoding for Spiking Transformers. That work sets per-position thresholds sinusoidally and cut energy 27.2% on language tasks, but it was never tried in control. We keep this secondary because exact decay may already suffice.
Finally, following Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access, we measure how much the timing tokens actually help predict returns. That paper showed that uninformative privileged inputs to a critic act as structured noise. A temporal encoding that adds capacity without adding predictive information should show up there as a cost.
Baselines
The naive reference point is the standard hybrid recipe on a jittered clock. It pairs a PopSAN actor with a twin-delayed deterministic policy gradient (TD3) multilayer-perceptron critic that sees only the current observation and action. It is run with identical environment steps, seeds, and actor architecture.
The strongest existing spiking-actor method for performance is MDC-SAN, from Multi-Sacle Dynamic Coding Improved Spiking Actor Network for Reinforcement Learning. It reports maximum average reward of 5628 on Ant-v3 and 11657 on HalfCheetah-v3, against 4671 and 10020 for a deep actor. Those numbers are on a fixed clock. No reported result was found on Lacuna for any spiking actor under irregular control intervals. We therefore reproduce MDC-SAN at fixed timing to confirm our setup is calibrated against those numbers, then run it under jitter.
The strongest baseline for the asymmetric-history idea itself is TD3BC+JSRL with an action-history critic, from the adaptive surrogate-gradient paper. It reports an average return of 400 in its quadrotor simulation, where vanilla TD3 and behavior cloning stay below −200.
The key ablations hold everything else fixed and change one thing in the critic. The first is an MLP critic with $\Delta t$ appended. The second is the same Transformer critic with integer positions, which is T-SAC transplanted onto a spiking actor. The third is our critic with timestamps randomly permuted within the window. All critics receive the same hyperparameter search budget, measured in GPU-hours, over learning rate, window length, and width. Parameter counts are matched between MLP and Transformer critics within 10%. Actor energy is reported for every run using spike counts in the NeuroBench-style accounting of the quadrotor paper, which reported 79% silent neurons.
Beating the baselines means a statistically separated improvement in final return across at least ten seeds under jitter, at no increase in actor spike rate. It also means our critic ties the baselines at a fixed clock.
Why not just…?
Why not just append $\Delta t$ to the observation of an MLP critic? This is the real threat. The file's argument against raw timestamps, drawn from trajectory and temporal-knowledge-graph work, is about generalizing over long absolute horizons, and that problem does not arise here. Our case rests on a different point. A critic that sees only the current state cannot know the actor's voltage, which was built up over past irregular gaps. The "MLP plus $\Delta t$" ablation tests this directly. If it matches our critic, especially when it is also given a fixed-length action history, we will report that the sequence encoding is unnecessary and that the contribution reduces to time-based discounting.
Why not just make the spiking actor's leak time-correct and leave the critic alone? Exact decay is cheap, and we adopt it in every condition, so it is not a competitor but a shared floor. What it does not fix is the critic's misspecification. Our decisive check compares jittered training with an exact-decay spiking actor under an MLP critic against the same actor under our critic.
Why not just resample sensor streams onto a fixed grid with zero-order hold? Holding the last observation removes jitter from the network's view, but it adds latency and discards the timing information the actor needs to react. We include a held-to-grid condition at matched decision rate and report both return and reaction latency.
Why not use a memoryless deep actor on an embedded GPU? The energy gaps reported for neuromorphic deployment are 140x for PopSAN on Loihi versus Jetson and 75x for SDDPG navigation (Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware). These are the motivation, not something we re-measure. We do train a memoryless deep actor with each critic, because it serves as the control for our mechanism.
Proposed Experiments
We will use MuJoCo Hopper-v3, Walker2d-v3, HalfCheetah-v3, and Ant-v3, modified so each decision interval is drawn from a log-normal distribution around the nominal step. Jitter is set at a coefficient of variation of 0, 0.25, 0.5, and 1.0, with the simulator integrating the held action for the drawn duration. The VMAS multi-agent sampling task is run with the interval varying between 0.1 and 0.01. CSAC's reported median returns there, 0.84–0.92 for continuous multi-agent SAC against 0.66 dropping to 0.43 for standard multi-agent SAC, give a calibration point for time-robustness with deep actors. The Crazyflie simulation setup of the adaptive surrogate-gradient paper, with jittered sensor timing, tests the warm-up-sensitive regime.
The decisive check is a two-by-two interaction. The first factor is the critic: our time-encoded Transformer versus MLP plus $\Delta t$. The second is the actor: a stateful spiking network versus a memoryless deep network. Both are measured across jitter levels. Our claim predicts three things. The critic gap should be near zero at no jitter. It should grow with jitter. It should be substantially larger for the spiking actor than for the memoryless one.
Supporting checks follow from the same runs. The permuted-timestamp ablation should erase the gain. A probe trained on the critic's hidden states should predict the actor's membrane voltages better than a probe on the MLP critic. Return Prediction Gain for the timing tokens should be positive exactly where the return gain appears. Threshold modulation in the actor is reported separately as a secondary result, judged on return and spike count. If the interaction is absent, the note's central claim fails, and we will say so.
References
- Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks
- Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns
- Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization
- Transformer Networks for Trajectory Forecasting
- Exploring Periodicity and Interactivity in Multi-Interest Framework for Sequential Recommendation
- Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control
- Positional Encoding for Spiking Transformers
- Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
- Multi-Sacle Dynamic Coding Improved Spiking Actor Network for Reinforcement Learning
- Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware