Scaling Recurrent Networks with Exponential Gating and Matrix Memory

This direction is merged into another direction: sequential processing bottleneck (sequential computation bottleneck). Its last version stays readable and citable here.

Version 1 of 112 papers19 conceptsfrozen 2026-10-04

For years, the Long Short-Term Memory (LSTM) network was the workhorse of sequence modeling. However, as the field moved toward the billion-parameter scale, LSTMs were largely eclipsed by Transformers and State Space Models (SSMs). The traditional LSTM faced two fundamental bottlenecks: the inability to revise memory once it was stored (due to sigmoid gating) and a limited storage capacity that scaled poorly with sequence complexity.

Recent work, led by Andreas Auer, Maximilian Beck, Sepp Hochreiter, and their colleagues, has revitalized this architecture through the Extended LSTM (xLSTM). By replacing traditional gates with exponential functions and introducing matrix-based memory, this new family of recurrent networks matches the performance of Transformers while maintaining the linear computational complexity of RNNs.

The Problem with Sigmoid Gating

In a classic LSTM, the "forget gate" uses a sigmoid function to decide what information to keep. Because the sigmoid function is bounded between 0 and 1, it acts as a restrictive filter. If the model encounters a piece of information that is significantly more important than what is currently in its memory, it cannot easily "overwrite" the old state; it can only gradually decay it.

As identified in xLSTM: Extended Long Short-Term Memory, this leads to a "vanishing influence" problem where the model struggles to reset its internal state or prioritize sudden, high-relevance inputs.

Exponential Gating and Memory Revision

The core innovation of the sLSTM (scalar LSTM) is the transition to exponential gating. Unlike sigmoid gates, exponential gates are unbounded. This allows the model to perform aggressive memory revision.

When a new, high-magnitude input arrives, the exponential gate can effectively "outweigh" the previous memory state. To prevent numerical instability (as exponentials can grow very large), these models introduce a normalizer state. This state tracks the sum of the gating weights, ensuring that the hidden state remains stable.

This mechanism is particularly powerful in domains with sudden shifts:

Expanding Capacity with Matrix Memory

While sLSTM improves how we update memory, the mLSTM (matrix LSTM) improves how much we can store. Traditional LSTMs store information in a vector. In contrast, mLSTM uses a matrix-based associative memory.

This shift transforms the recurrent update into something resembling the "Key-Value" lookups found in Transformer attention, but performed recurrently. This allows the model to store more complex relationships without increasing the hidden state's dimensionality to prohibitive levels.

In the context of Graph Neural Networks, Alvaro Arroyo, Hugh Blayney, and Michael M. Bronstein use this matrix memory in gLSTM to solve the "over-squashing" problem. Over-squashing occurs when a node in a graph fails to compress information from a large neighborhood into a fixed-size vector. By using matrix memory, gLSTM provides the "room" necessary to store signals from distant neighbors.

A Generic Backbone for Vision and Audio

The efficiency of xLSTM—specifically its ability to process sequences with linear complexity while maintaining high expressive power—has led to its adoption as a general-purpose backbone.

In Vision-LSTM: xLSTM as Generic Vision Backbone, Benedikt Alkin and Johannes Brandstetter adapt the architecture for images. By treating image patches as a sequence and using an alternating "flip" traversal strategy, they achieve results that outperform Vision Transformers (ViTs) on ImageNet. Similarly, Sarthak Yadav demonstrates in AxLSTMs that xLSTM can learn superior self-supervised audio representations with significantly fewer parameters than Transformer-based counterparts.

Why it Works: The Intuition

The success of the xLSTM family stems from a shift in how we view recurrent memory. Traditional RNNs were "leaky buckets"—they tried to hold onto everything but inevitably lost information as new data poured in.

The xLSTM architecture turns the bucket into a dynamic ledger:

  1. Exponential Gating provides the "eraser" and "bold pen," allowing the model to strike out irrelevant history and highlight critical new information instantly.
  2. Matrix Memory provides the "pages," expanding the storage space so that different types of information don't have to compete for the same few scalar slots.
  3. Parallelization (in mLSTM) allows these updates to happen across the sequence simultaneously during training, removing the primary speed disadvantage that once made LSTMs uncompetitive with Transformers.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_recurrent_networks_with_exponent_3e9c09c0,
  title = {Scaling Recurrent Networks with Exponential Gating and Matrix Memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-recurrent-networks-with-exponential-gating-and-matrix-memory/rdr_3e9c09c08b8145c98788d0079cb5feee},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_recurrent_networks_with_exponent_3e9c09c0_v1,
  title = {Scaling Recurrent Networks with Exponential Gating and Matrix Memory},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-recurrent-networks-with-exponential-gating-and-matrix-memory/rdr_3e9c09c08b8145c98788d0079cb5feee/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}