Recurrent models that store context in a matrix instead of a vector can remember far more. The matrix-memory LSTM, or mLSTM (xLSTM: Extended Long Short-Term Memory), writes each token into a $d \times d$ table as the outer product of a "key" and a "value" vector. Later it reads that table back with a "query" vector. This lets it handle multi-query associative recall, which asks a model to store many key–value pairs and retrieve them on demand, and on which vector-state recurrent models fail. The cost is that every token touches all $d^2$ entries with floating-point multiplies. Spiking neural networks take the opposite trade. Their neurons emit sparse discrete pulses, so a weight is either added, subtracted, or skipped, and work is done only when a pulse fires. But the spiking sequence models built so far, such as SpikeGPT, keep vector-sized states and inherit the recall deficit that vector states have.
We propose a matrix memory whose cost tracks pulse activity rather than matrix size. Keys, values and queries are sparse ternary vectors, with entries in $\{-1,0,+1\}$. A write then touches only the matrix entries where both the key and the value fire, and a read touches only the columns where the query fires. The one remaining dense operation is the forget gate, which shrinks the whole matrix every step. We remove it by folding the gate into a single deferred per-head scale, quantized to powers of two so that rescaling is a bit shift. A short argument, given below, predicts that sparsifying random codes costs nothing in recall capacity while reducing write work by the product of the key and value firing rates. The proposal's central test is whether that holds for learned codes on associative recall benchmarks.
Core Claim
Take a single mLSTM head with scalar input gate $i_t$ and scalar forget gate $f_t$. It updates $M_t = f_t M_{t-1} + i_t\, v_t k_t^\top$ and reads $y_t = M_t q_t$, normalized by a running key sum. Let $F_t = \prod_{\tau \le t} f_\tau$ be the cumulative forget factor. Then the memory can be stored as $M_t = F_t \tilde M_t$, with the stored matrix $\tilde M_t$ updated as
$$\tilde M_t = \tilde M_{t-1} + \frac{i_t}{F_t}\, v_t k_t^\top .$$
This rewrite is exact. If $k_t$ and $v_t$ are ternary with firing rates $\rho_k$ and $\rho_v$, the update adds or subtracts one per-step scalar at about $\rho_k \rho_v d^2$ entries and multiplies nothing. If the query is ternary with rate $\rho_q$, the read $\tilde M_t q_t$ costs about $\rho_q d^2$ additions, followed by one scalar rescale per head.
The factor $F_t$ shrinks toward zero over time, so $\tilde M_t$ must be renormalized every so often. We restrict $\log_2 f_t$ and $\log_2 i_t$ to a learned integer grid. That makes $F_t$ a power of two, renormalization becomes an exponent shift, and the exponential gating that xLSTM relies on for revising memory survives without exponentials or multiplies at inference.
The capacity prediction comes from a crosstalk count. Store $N$ pairs whose keys are random sparse ternary vectors, each with $\rho_k d$ nonzeros. Retrieving one key gives a signal of $\rho_k d$. Each other stored key overlaps it in about $\rho_k^2 d$ random-sign positions, so the summed interference has standard deviation of about $\rho_k\sqrt{Nd}$. The signal-to-noise ratio is therefore about $\sqrt{d/N}$, the same as for dense codes and independent of $\rho_k$ as long as $\rho_k d \gg 1$. In principle, then, sparsity is free for capacity and cuts write work by $\rho_k\rho_v$. Whether learned keys behave like random ones is the empirical question.
Relation to Prior Work
Three papers motivate the gap. Zoology (Zoology: Measuring and Improving Recall in Efficient Language Models) shows that most of the perplexity gap between efficient models and Transformers comes from associative-recall tokens. It also shows that a 70M attention model out-recalls a 1.4B Hyena model. xLSTM shows that matrix memory with exponential gating beats Mamba and RWKV-4 at 1.3B parameters and improves MQAR, the multi-query associative recall benchmark. The RWKV survey (A Survey of RWKV) documents RWKV moving to matrix states in versions 5 through 7, while noting that only the vector-state RWKV-4 was ever made spiking.
The spiking literature supplies techniques and also constrains the design. SpikeGPT (SpikeGPT) trains a spiking RWKV-4 with arctangent surrogate gradients, a smooth stand-in for the step function's zero gradient. We reuse that training recipe but replace its vector state with the event-driven matrix. Bipolar self-attention (Bipolar Self-attention for Spiking Transformers) shows two things. Binary spikes lose sign information, which motivates our ternary codes. Exponentials are too costly for spiking hardware, which motivates our power-of-two gates in place of xLSTM's exponentials. That work applies to quadratic vision attention and has no recurrent state.
NSLLM (Neuromorphic spike-based large language model) converts pretrained LLMs into integer-spiking linear attention on an FPGA. It is the strongest existing evidence that multiplier-free linear attention is useful, but it uses multi-bit integer neurons obtained by conversion rather than sparse ternary codes trained for sparsity. LRMM (LIF Recurrent Memory Enables Long-Horizon Spiking Computation) and the harmonic resonate-and-fire state space model (A Spiking Heterogeneous Harmonic Resonate-and-Fire State Space Model for Time Series) show that spiking recurrence can hold information over tens of thousands of steps. Their states are vector-sized, and they were never tested on multi-query recall. 3DSMT (3DSMT) and the same harmonic model also caution that operation counts do not become real energy savings on GPUs. We take that seriously in the evaluation.
A clarification on terminology: "spiking" here means one ternary event vector per token, as in SpikeGPT. It does not mean multi-timestep rate coding. The claim is about event-proportional work, not biological fidelity.
Research Program
We will build the cell on top of the official xLSTM code (NX-AI/xlstm). The key, value and query projections feed ternary threshold neurons with learnable thresholds. Gradients pass through them via a triangle surrogate. A firing-rate penalty sets $\rho_k$, $\rho_v$ and $\rho_q$ to target levels.
Training uses the standard parallel chunkwise mLSTM kernel, because the deferred-scale form is mathematically identical. That avoids the cost of unrolling backpropagation through time that afflicts most spiking training. Gate log-values are rounded to integers with straight-through estimation. The normalizer divisor is rounded to a power of two. Block outputs pass through ternary neurons before the next layer, so the whole stack communicates by events.
A separate inference engine implements the sparse write and read literally and logs every accumulate, shift and state-memory access. Its job is to confirm that the counted operations match the measured nonzeros, and that the renormalization schedule never overflows a fixed-point store of chosen bit width.
The analysis has two strands. First, we measure how far learned keys depart from the random-code capacity prediction by tracking pairwise key overlaps during training. Second, we characterize how bit width, sparsity and recall trade against each other.
Baselines
The naive reference point is a matched-state spiking vector recurrence: SpikeGPT's spiking RWKV-4 block at the same width and depth, trained identically. We will also run LRMM in its published configuration on the spiking benchmarks. LRMM reports 96.52% on PS-MNIST, a long-horizon digit task in which image pixels are fed one at a time in permuted order. If our cell does not beat these models on MQAR once the number of stored pairs exceeds the vector width, the matrix adds nothing.
The strongest existing method on the question is the floating-point mLSTM at the same head dimension, layer count, token budget and learning-rate sweep. It is the accuracy ceiling. No reported MQAR number is found on Lacuna for xLSTM. The file also notes that Mamba and Hyena reach near-perfect MQAR only with careful learning-rate tuning (Revisiting Associative Recall in Modern Recurrent Models). We will therefore give every baseline the same tuning grid. Beating the ceiling is not the goal. Success means staying within a few points of its MQAR accuracy and language-model perplexity while counting at least an order of magnitude fewer state operations per token.
On spiking audio, the reference numbers come from The Heidelberg Spiking Data Sets, which reports LSTM accuracy of 85.7% on SHD and 75.0% on SSC. It also reports recurrent spiking network accuracy of 71.4% on SHD and 50.9% on SSC, and 83.2% on SHD for the optimized version with augmentation.
The ablation that isolates the key ingredient is a dense ternary mLSTM. It has the same ternary keys and values but no sparsity target, so firing rates settle wherever training leaves them, and its gates are continuous. It separates the effect of sparsity control and power-of-two gating from the effect of ternarization alone. A second ablation keeps sparsity but uses continuous gates, which isolates what quantizing the gates costs.
Why not just…?
Why not just use an existing spiking vector model such as the harmonic resonate-and-fire model or LRMM? These models handle very long horizons cheaply. But Zoology's analysis implies that a fixed vector state can store only about as many associations as its width. We test this directly: we sweep the number of key–value pairs in MQAR past $d$ at matched total state bytes. If the vector models keep pace, our motivation collapses, and the experiment is designed to show that.
Why not just ternarize a pretrained mLSTM after training, NSLLM-style? This is a real threat, because conversion avoids surrogate-gradient instability entirely. We will convert an mLSTM trained at the same budget using NSLLM's sensitivity-guided integer quantization. We compare accuracy at matched counted operations. Our expectation is that conversion keeps accuracy only with multi-bit, dense activations, and loses the $\rho_k\rho_v$ saving that requires codes trained for sparsity. If conversion matches us at similar sparsity, we will report that the deferred-scale engine is the real contribution and training-time sparsification is not needed.
Why not just add a few attention layers to a spiking vector model, as Zoology's hybrids do? Zoology reports that under 10% sparse attention closes 97.4% of the perplexity gap. The catch is that attention layers bring back a key–value cache that grows with context. That breaks the fixed-memory property needed for streaming on neuromorphic chips. We will include a spiking hybrid with one attention layer and report both accuracy and the per-token state memory at 2k and 16k context. If the hybrid wins at short context but its memory grows linearly, the comparison is settled by deployment constraints rather than accuracy.
Proposed Experiments
The decisive experiment is MQAR with a vocabulary of 8,192 tokens, following the Revisiting setup. We sweep the number of stored pairs, the sequence length up to 4k tokens, and target firing rates from dense down to about 5%. The theory predicts a flat recall-versus-sparsity curve down to $\rho_k d$ of a few tens. If recall instead degrades steadily, learned keys are not behaving like random codes, and we will report where the trade-off sits.
To connect with the spiking literature, the same cell is trained on PS-MNIST, the copy task, SHD and SSC against LRMM and the Heidelberg reference numbers. For language modeling, we train 46M-parameter models on Enwik8 and roughly 200M-parameter models on OpenWebText2 next to SpikeGPT and the float mLSTM, and report bits per character, perplexity and recall-token loss. No reported SpikeGPT bits-per-character or perplexity figure is found on Lacuna, so SpikeGPT will be retrained under our pipeline.
Energy is reported three ways: logged accumulate, shift and memory-access counts from the sparse engine; energies derived from those counts using the per-operation constants in the Spikingformer energy scripts (TheBrainLab/Spikingformer); and, as a stretch goal, wall-clock power for the sparse write and read mapped onto an FPGA in the style of NSLLM. Because state-memory traffic may dominate energy more than arithmetic does, every count is split into arithmetic and memory access. The claim succeeds only if both shrink with $\rho_k\rho_v$ and $\rho_q$ as the formulas predict.
References
- xLSTM: Extended Long Short-Term Memory
- Zoology: Measuring and Improving Recall in Efficient Language Models
- A survey of RWKV
- SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks
- Bipolar Self-attention for Spiking Transformers
- Neuromorphic spike-based large language model
- LIF Recurrent Memory Enables Long-Horizon Spiking Computation
- A Spiking Heterogeneous Harmonic Resonate-and-Fire State Space Model for Time Series
- 3DSMT: A Hybrid Spiking Mamba-Transformer for Point Cloud Analysis
- Revisiting Associative Recall in Modern Recurrent Models
- The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks