When an overparameterized recurrent network is trained by gradient descent, the optimizer does not pick an arbitrary solution among those that fit the data. It prefers some temporal patterns over others. For linear recurrent networks this preference has been computed exactly. In the infinite-width limit, training behaves like fitting a one-dimensional convolution whose weight on an input $j$ steps in the past is scaled by a factor that shrinks geometrically in $j$. The result is a built-in bias toward recent inputs, which helps when data are scarce and hurts on long delays (Implicit Bias of Linear RNNs). Biological neurons carry a second, slower state variable: an adaptation current, or equivalently a rising firing threshold, that grows after each spike and suppresses later firing. Adaptation is known empirically to rescue spiking networks on long-memory tasks. On sequential MNIST, adding it lifts a recurrent spiking network from 51.8% to 93.7% (Spike frequency adaptation supports network computations on temporally dispersed information). Nobody has asked what adaptation does to the optimizer's temporal preference.
We propose to derive and test the implicit temporal prior of recurrent networks built from adaptive neurons. Our central prediction is that adaptation does not simply lengthen memory. Because it is negative feedback triggered by the neuron's own output, it should reshape the optimizer's preferred temporal filter into a signed, band-pass profile: positive weight on recent inputs, and a slow negative tail that encodes the past as a reduction of response. This is the gradient-level counterpart of the "negative imprinting" that Salaj et al. observed after training. The prediction separates adaptation from the obvious alternative of adding slow leaky units, which should yield a positive, low-pass long tail. The two mechanisms should therefore win on different tasks, and that is directly testable.
Core Claim
Consider a network of adaptive neurons linearized around a steady firing rate. Each neuron has a fast membrane variable $v_t$ and a slow adaptation variable $a_t$:
$$v_t = \alpha v_{t-1} + \tfrac{1}{\sqrt{n}}(W r_{t-1})_i + (F x_t)_i - a_t,\qquad a_t = \beta a_{t-1} + \kappa\, r_{t-1}.$$
Here $r_t$ is the neuron's output rate (linearized spiking), $\alpha$ and $\beta$ are the membrane and adaptation decay factors, and $\beta$ is close to 1. $\kappa \ge 0$ is the adaptation strength, playing the role of the spike-triggered increment $b$ in the Adaptive Exponential Integrate-and-Fire model (Firing patterns in the adaptive exponential integrate-and-fire model). $W$ is the recurrent weight matrix, $F$ the input weights, and $n$ the width.
In isolation, each neuron responds to an impulse with a difference of exponentials that changes sign. We conjecture, and will try to prove by extending the infinite-width tangent-kernel argument of Emami et al., that the equivalent convolution's lag-scaling factors take the form
$$\rho_j = \sum_{k\ge 0} \nu_W^{k}\, c_k(j;\alpha,\beta,\kappa).$$
In this expression, $\nu_W$ is the recurrent-weight variance at initialization, and $c_k$ is the contribution of paths that pass through $k$ recurrent hops, fixed by the neuron's own impulse response. Setting $\kappa=0$ and $\alpha=0$ must recover the geometric profile $\rho_j \propto \nu_W^{j-1}$. The substantive claims are two. First, for $\kappa>0$ the profile acquires a slowly decaying tail governed by $\beta$ rather than $\nu_W$, so long memory no longer requires pushing $\nu_W$ toward the instability edge. Second, the effective filter's structure is signed. Under a facilitation variant ($\kappa<0$), or a matched slow leaky unit, the tail should have the opposite sign.
Research Program
The work has three layers.
The analytic layer derives the lag-scaling factors for the linearized adaptive network in the infinite-width limit. It computes them numerically for grids of $(\alpha,\beta,\kappa,\nu_W)$ and checks them against empirical tangent kernels of finite-width rate networks.
The spiking layer asks whether the prediction survives the real nonlinearity. There, threshold crossings are non-differentiable, and training relies on surrogate derivatives, whose effect on optimization is itself uncharacterized. We train generalized leaky integrate-and-fire networks with adaptive thresholds, as in Salaj et al., using backpropagation through time with surrogate gradients. We measure the network's empirical temporal sensitivity kernel, the gradient of the output with respect to the input at lag $j$. We measure it at initialization and after training, and compare its shape and sign to the prediction. We sweep surrogate width, because it sets the operating-point gain in the linearization.
The third layer tests whether the prior matters for generalization in low-data regimes. If the analysis is right, the optimizer's filter preference should predict which tasks adaptation helps and which it hurts.
This extends On the Implicit Bias of Gradient Descent for Temporal Extrapolation, which shows that gradient descent on linear recurrent systems mutes unused eigenmodes and supports length extrapolation. That analysis has only one kind of state and no self-inhibition. We add a second state that feeds back negatively on output. A Brain-Inspired Gating Mechanism Unlocks Robust Computation in Spiking Neural Networks studies activity-dependent conductance for stability and noise rejection, not for what gradient descent prefers. We treat its gated neuron as a secondary test case, since input-dependent decay breaks our linearization. As a smaller aside, we will check whether the linear-memory online rule of Model-agnostic linear-memory online learning in spiking neural networks, which approximates the recurrent Jacobian as diagonal, preserves the adaptation-induced tail. That approximation drops exactly the cross-neuron paths in $c_k$ for $k\ge1$.
Baselines
The naive reference is a recurrent network of non-adaptive leaky integrate-and-fire neurons, plus its rate-based counterpart, Emami et al.'s linear recurrent network. This is the setting where the geometric short-memory prior is proven. Salaj et al. report 51.8% on sequential MNIST for this spiking variant, against 93.7% with adaptation. We will rerun both rather than rely on the reported gap.
The strongest mechanistic competitor is a timescale-matched slow leaky unit. We give each neuron an extra positive, low-pass state with decay $\beta$, or use heterogeneous membrane time constants spanning the same range. This competitor has the same number of slow variables and the same memory horizon, so any difference isolates the negative-feedback sign.
The key ablation is facilitation: the same adaptive neuron with $\kappa$ flipped negative, so that spikes lower rather than raise the threshold. It keeps every architectural detail and changes only the sign that our theory says matters.
For accuracy context on real spike data, the strongest numbers in the file are on Spiking Heidelberg Digits. The online-trained spiking network in the BrainTrace paper reports 95.33%, the dynamic gated neuron reports 88.98%, an LSTM reports 85.7%, and a plain recurrent spiking network reports 71.4% (The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks). We also include an LSTM and an explicitly regularized non-adaptive network (weight decay, and a penalty on the tail of the learned input filter).
All models share the same data splits, width, parameter count within 5%, training steps, and an equal random-search budget over learning rate and initialization scale. Each result is averaged over at least five seeds.
"Beating" a baseline is not the main criterion here, because this is a claim about mechanism. We succeed if the predicted $\rho_j$ matches empirical sensitivity kernels in rate networks (rank correlation above 0.9) and matches them in sign and tail timescale in spiking networks. We also succeed if adaptation and slow-leak networks show the predicted crossover in low-data generalization, with facilitation behaving like slow-leak. Failing to match the reported full-data Spiking Heidelberg Digits numbers would not falsify the claim; a missing crossover would.
Why not just…?
Why not just add slow leaky units, since any slow state would lengthen memory? This is the most serious threat. If slow leaky units produced the same temporal prior as adaptation, the biological mechanism would be irrelevant to the question. Our theory says they should differ in sign, not in horizon. The matched slow-leak baseline and the facilitation ablation exist to settle this. If all three give indistinguishable kernels and indistinguishable crossovers, we will report that adaptation's benefit reduces to "any slow variable" and drop the band-pass claim.
Why not just train the networks and look at the learned kernels, without theory? Post-hoc inspection is how negative imprinting was found in the first place (Salaj et al.). It cannot say whether the pattern comes from the data or from the optimizer. Following the logic of Cohen-Karlik et al., we use memoryless-teacher tasks whose targets have no temporal structure. Any lag structure that appears is then attributable to the optimizer. A theory that predicts that structure from $(\alpha,\beta,\kappa)$ before training is the added value.
Why not just impose the desired filter with explicit regularization or orthogonal recurrent weights? Explicit penalties need a target filter shape known in advance and a tuned strength. Emami et al. show that the architectural prior already regularizes in low-data regimes without tuning. Orthogonal recurrence removes decay but has only one sign and cannot express the negative slow tail. We include the explicitly regularized baseline under the same tuning budget. If it matches adaptation everywhere, the practical case for the prior weakens, though the descriptive result stands.
Why trust an infinite-width, linearized analysis for finite spiking networks trained away from the kernel regime? This is a genuine limitation. Diagonal-network analyses show that initialization scale moves training between kernel-like and feature-learning behavior (Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization). We therefore sweep width and initialization scale, and report where the predicted kernel stops matching. We expect the sign structure to be more robust than the exact profile.
Proposed Experiments
The first experiment is synthetic. It uses Emami et al.'s delay-regression setup with delay $\tau$, extended with two target families. Integration targets have a positive long filter. Change-detection targets have a filter equal to a recent input minus a slowly weighted past. Training sets range from 50 to 5,000 sequences. We measure test $R^2$ as a function of $\tau$ and training-set size, along with the fit between measured and predicted kernels. The decisive check is the crossover. At small data, adaptive networks should beat slow-leak networks on change detection and lose to them on integration, and facilitation should track slow-leak. At large data, the gap should shrink, because a prior matters most when data are scarce.
The second experiment is STORE-RECALL and 12AX, where Salaj et al. report above 99% up to 1.6-second delays and 97.79% for adaptive networks, against chance and 0.39% for plain ones. We run these under reduced training budgets, with memoryless-teacher controls to confirm the optimizer-origin of the kernel.
The third experiment uses sequential MNIST, Spiking Heidelberg Digits, and Spiking Speech Commands, reporting accuracy and sensitivity kernels for all baselines at 10%, 30%, and 100% of the training data. For Spiking Speech Commands, the cited reference points are 75.0% for an LSTM and 50.9% for a plain recurrent spiking network; no reported result was found on Lacuna for adaptive networks on it. For the macaque somatosensory recordings used by Emami et al., no numeric $R^2$ value is reported on Lacuna. If we can obtain the data, we will use it as a neural-data test of the low-data prediction.
The final check repeats the synthetic crossover with the diagonal-Jacobian online rule. The analysis predicts that this rule retains each neuron's own adaptation tail but loses the recurrent contributions. The adaptation-driven advantage should therefore survive when the task is solvable one neuron at a time and shrink when it requires cross-neuron memory.
References
- Implicit Bias of Linear RNNs
- Spike frequency adaptation supports network computations on temporally dispersed information
- Firing patterns in the adaptive exponential integrate-and-fire model
- On the Implicit Bias of Gradient Descent for Temporal Extrapolation
- A Brain-Inspired Gating Mechanism Unlocks Robust Computation in Spiking Neural Networks
- Model-agnostic linear-memory online learning in spiking neural networks
- The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks
- Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization