Outcome-Indexed Eligibility Traces for Stochastic Spiking Networks

When a spiking network earns a reward seconds after the spikes that caused it, the brain-inspired learning rules we have give credit to whatever fired recently. A reward-modulated learning rule keeps a decaying memory, called an eligibility trace, at each synapse, noting "this input and this output were active together." When a reward arrives, that trace is multiplied by the reward. If the reward comes after the trace has decayed, the credit is lost. If the trace is made long enough to survive the delay, every spike in the window receives credit, and the learning signal fills with noise. This note proposes a way to avoid that trade-off for networks whose neurons fire probabilistically. Each synapse keeps one trace for each possible outcome, for example "success" and "failure." Each spike is written into each trace weighted by how much more (or less) likely that spike would be if we knew the episode ended in that outcome. When the real outcome arrives, the synapse keeps the matching trace and discards the others. Credit then depends on whether a spike predicts the outcome, not on how recently it happened. Memory scales with the number of outcomes times the number of synapses, not with episode length.

The mechanism combines two existing ideas. The first is hindsight credit assignment from reinforcement learning, Hindsight Credit Assignment. It weights each past action by the ratio between the probability of that action given the eventual outcome and its ordinary probability, and this ratio does not decay with elapsed time. The second is stochastic spike response modeling. A neuron's spike is a coin flip whose probability depends smoothly on its membrane potential, as in the escape-noise neurons of Associative Memory in a Network of 'Biological' Neurons. Such a neuron is exactly a tiny stochastic policy, so the hindsight ratio is well defined for every spike. The two ideas do not fit together naively, because the ratio needs the outcome and the outcome arrives only at the end. For a finite set of outcomes, that obstacle disappears: one trace is accumulated per candidate outcome, and the correct one is selected when the reward arrives.

Core Claim

In a recurrent network of escape-noise neurons, hindsight credit for a delayed, discrete outcome can be computed with online, synapse-local accumulators. The resulting weight update has variance well below REINFORCE with a reward-prediction-error baseline. REINFORCE here means the standard unbiased estimator that multiplies each spike's score by the reward. Its effectiveness should be nearly flat as the delay grows, where decaying-trace rules degrade.

Concretely, neuron $i$ fires at step $t$ with probability $p_i(t)=\sigma(V_i(t)-\theta)$, where $V_i$ is its membrane potential, $\theta$ its threshold, and $\sigma$ the logistic function. We define a hindsight model $h_{i,k}(t)=\sigma(V_i(t)-\theta+c_{i,k})$. It predicts the same neuron's firing probability if the episode is known to end in outcome $k$, using a learned offset $c_{i,k}$ per neuron and outcome. This offset can optionally be made to depend on a coarse task-phase index.

Writing $\pi_i(s)$ and $h_{i,k}(s)$ for the probability each model assigns to the sampled spike value $s\in\{0,1\}$, synapse $j\to i$ keeps one accumulator per outcome:

$$E^{k}_{ij}=\sum_{t}\big(s_i(t)-p_i(t)\big)\,x_j(t)\Big(1-\frac{\pi_i(s_i(t))}{h_{i,k}(s_i(t))}\Big).$$

Here $x_j(t)$ is the filtered presynaptic input. The first two factors are the exact score function of the Bernoulli spike, and the last factor is the return-conditional hindsight weight from HCA.

At episode end, with realized outcome $Z$ and return $G$, the update is $\Delta w_{ij}=\eta\,G\,E^{Z}_{ij}$, where $\eta$ is the learning rate. The offsets are fit by maximum likelihood on realized spikes. Their gradient, $\sum_t (s_i(t)-h_{i,Z}(t))$, can also be accumulated online per outcome.

If a neuron's firing carries no information about the outcome, its hindsight model converges to $h\approx\pi$ and its update vanishes. This is the intended source of variance reduction: most spikes in a large recurrent network are irrelevant to any particular reward.

Relation to Prior Work

Three existing papers join these two directions, and each stops short of this proposal.

Functional Requirements for Reward-Modulated Spike-Timing-Dependent Plasticity uses the same escape-noise neurons and proves two things. Such rules are only goal-directed when the modulatory signal is a stimulus-specific reward prediction error. Credit is carried by forward-decaying traces. We adopt its critic requirement as a constraint, and we replace the decaying trace with outcome-indexed accumulators.

Hindsight Network Credit Assignment applies hindsight reasoning to networks of Bernoulli units. It does so across layers, in static bandit and variational autoencoder settings, and explicitly leaves recurrent temporal dynamics open. Our hindsight ratio is taken over time and conditioned on the outcome. HNCA's ratio is taken over a unit's immediate children at one instant. The two could compose, and one ablation tests exactly that.

Credit Assignment Techniques in Stochastic Computation Graphs supplies the theory that licenses conditioning credit on sets of nodes that screen off the loss. It proposes learned gradient-critics. Our design deliberately avoids an auxiliary critic network: the hindsight model is a single offset per neuron and outcome.

Other attempts to bridge long delays in spiking networks keep time contiguity and stretch it. Theta sequences as eligibility traces compresses behavior so that short traces span longer delays. Diffusion of Neuromodulators for Temporal Credit Assignment spreads sparse feedback spatially but still relies on e-prop's decaying traces, where e-prop (eligibility propagation) is an online approximation to backpropagation through time. Our proposal is motivated by the survey A Survey of Temporal Credit Assignment in Deep Reinforcement Learning, which identifies depth (long delays) and breadth (many confounding paths) as the axes where recency-based credit fails. A recurrent spiking network is extreme on breadth.

Research Program

The first strand is analytic. We will show that the estimator is unbiased whenever $h_{i,k}$ equals the true outcome-conditional spike probability, which follows from HCA's return-conditional theorem applied to each neuron as a policy. We will then bound the bias introduced by the offset-only hindsight model as a function of its likelihood gap. We will also derive the variance reduction relative to REINFORCE with a baseline in a linear-Gaussian toy network, where the fraction of outcome-informative neurons is controlled.

The second strand is engineering. We will build discrete-time recurrent escape-noise networks with adaptive thresholds, implement the $K$-accumulator rule with clipped ratios, and add the stimulus-specific critic required by the R-STDP analysis. Memory is $K$ times the number of synapses, plus $K$ offsets per neuron.

The third strand is diagnosis. On tasks with known causal structure, we will measure two things. The first is the cosine similarity between each estimator's update and a reference gradient. That reference comes from very-high-sample REINFORCE or from exact expected-value backpropagation where that is tractable. The second is gradient variance on a log scale, matching the metric HNCA reports.

Baselines

The naive reference is a reward-modulated rule with a decaying eligibility trace and a global reward baseline. This is the configuration shown by Functional Requirements for Reward-Modulated STDP to suffer from unsupervised drift. It is the floor any method must clear.

The strongest spiking-native competitor is the same rule with a stimulus-specific critic, with its trace time constant tuned per delay under the same search budget we give our learning rate and clipping threshold. As the time constant goes to infinity, this becomes REINFORCE with a learned baseline. The sweep therefore covers both ends of the time-contiguity family.

A second strong competitor is e-prop with sparse diffusive feedback, following Diffusion of Neuromodulators for Temporal Credit Assignment. That paper reports it significantly outperforming standard e-prop on delayed match-to-sample and 7-cue accumulation, but the file gives no scalar accuracy for those tasks. It is run on identical network sizes and tasks.

The supervised upper reference is surrogate-gradient backpropagation through time on a deterministic leaky integrate-and-fire (LIF) network, as in STCA. It receives dense labels rather than delayed scalar reward. For the bandit-feedback Spiking Heidelberg Digits (SHD) task we compare against the supervised numbers in the file. These are 71.4% for a single-layer recurrent spiking network and 83.2% with augmentation (The Heidelberg spiking datasets), and 92.36% for HetSynLIF (HetSyn). No reported result was found on Lacuna for SHD under delayed bandit reward, so we establish our own reward-trained baselines there.

On gradient variance, the anchor is HNCA on one-layer Contextual Bandit MNIST. It reports log gradient variance of about −13.0 for HNCA against about −6.0 for REINFORCE.

The key ablation replaces the learned $h$ with an outcome-shuffled $h$ to confirm that the benefit comes from outcome information. A second ablation composes our temporal ratio with HNCA's child-based ratio. Beating the tuned-trace baseline means reaching the same success rate in fewer episodes at every delay above the trace's best time constant, with lower measured variance. A single matched delay is not enough.

Why not just…?

Why not just use a longer eligibility trace? This is the real threat. In the limit it is REINFORCE with a critic, which is unbiased and simple. Our claim is specifically that the hindsight factor removes the variance that long traces accumulate from outcome-irrelevant spikes. The time-constant sweep settles it directly. If the best-tuned trace matches our variance and sample efficiency at long delays, the idea has failed, and we will report it as such.

Why not just train with surrogate-gradient backpropagation through time? It needs dense differentiable losses or a differentiable model of the environment, and memory that grows with episode length. P-SpikeSSM documents LIF backpropagation through time at 6.1 minutes per epoch versus 1.5 for parallel probabilistic sampling on permuted sequential MNIST (psMNIST). It also cannot consume a scalar reward from a non-differentiable environment without an added critic. We treat it as a supervised ceiling, not a competitor on reward learning.

Why not just apply HNCA at every time step? HNCA's ratio looks only at a unit's immediate children and uses the reward as the multiplier. It reduces spatial variance but leaves the long temporal chain between spike and reward untouched. The composition ablation tests whether the two ratios are complementary or whether HNCA alone already captures most of the gain.

Why not just redistribute reward with a transformer, as in CoDeTr? Beyond Simple Sum of Delayed Rewards does this well in dense floating-point networks, but it stores whole trajectories and runs attention over them. Our goal is an online rule with memory independent of episode length. We will report the memory and compute trade-off rather than claim superior returns.

Proposed Experiments

We start with the HCA diagnostics from Hindsight Credit Assignment, Delayed Effect and Ambiguous Bandit, implemented with a spiking policy network so that ground-truth causal structure is known.

We then build a Delayed Contextual Bandit MNIST: the HNCA task with a silent recurrent delay of $D$ steps inserted between stimulus and reward. We sweep $D$ from 10 to 1000.

Next come cue accumulation and delayed match-to-sample, with reward only at trial end, as used in the diffusion and HetSyn papers. Finally, SHD with reward given once per utterance (1 if the class is correct, 0 otherwise).

Metrics are success rate against episodes, log gradient variance, and cosine similarity to the reference gradient, each reported as a function of $D$.

The decisive check has two parts. The learning curve for outcome-indexed traces should stay nearly flat as $D$ grows while the best tuned-trace baseline degrades. The shuffled-hindsight ablation should collapse to or below REINFORCE-with-critic performance. If both hold, the gain is attributable to outcome-conditioned credit rather than to tuning. A secondary check is bias: whether the offset-only hindsight model's likelihood gap predicts the residual gap to the reference gradient.