Structured Matrix Duality in Sequence Modeling

22 concepts18 papers1 proposal

For years, the field of sequence modeling has been defined by architectural fragmentation. On one side stand Transformers, which dominate language modeling through the expressive power of quadratic ($O(N^2)$) self-attention. On another side sit Recurrent Neural Networks (RNNs) and State Space Models (SSMs), which offer linear ($O(N)$) time inference and stable memory footprints but have historically struggled to match the performance or training parallelizability of attention.

As noted by Carmen Alonso et al. and Kazuki Irie et al., developing these architectures in isolation has obscured a deeper mathematical reality. Recent work has begun bridging this cross-architectural gap, revealing that attention, SSMs, and RNNs are not fundamentally distinct paradigms. Instead, they are different facets of a unified framework bound together by the mathematics of structured matrices and linear operators.

Sequence Mixers as Structured Matrices

To understand why these architectures can be unified, we must first view them through a shared lens: the matrix mixer. Any sequence modeling layer takes an input sequence of tokens and produces an output sequence by mixing information across different time steps.

As formalized by Sukjun Hwang et al., this sequence-mixing operation can be generalized as a linear transformation where the input sequence matrix is multiplied by a highly structured mixing matrix $M$:

$$Y = M X$$

The differences between modern architectures lie entirely in the structural constraints and properties imposed on this matrix $M$:

  • Standard Attention: $M$ is a dense, data-dependent softmax matrix. Because every token can look at every prior token without restriction, computing and storing $M$ requires quadratic $O(N^2)$ time and space.
  • Linear Attention: The softmax operation is removed or replaced by a feature map, allowing $M$ to be factored into a low-rank product of keys and values, dropping the computational cost to $O(N)$.
  • State Space Models (SSMs): Information is passed sequentially through a hidden state via a linear recurrence relation. This sequential constraint forces $M$ to take the form of a semiseparable or quasiseparable matrix, where elements below the diagonal are structured by the decay of the recurrence.

By treating sequence mixing as a structured matrix operation, researchers can directly map the behavior of one architecture onto another. For instance, Aviv Bick et al. leveraged this perspective to develop MOHAWK, a framework that distills the quadratic attention matrices of a pretrained Transformer teacher into the subquadratic structured matrices of an SSM student, allowing efficient models to inherit the capabilities of Transformers with minimal training data.

The Core Equivalence: State Space Duality

The most profound realization of this unification is State Space Duality (SSD), discovered by Tri Dao et al.. The authors proved a formal mathematical equivalence between selective State Space Models (like Mamba) and linear attention mechanisms.

To build intuition for why this works, consider how a linear SSM updates its hidden state $h_t$ given an input $x_t$:

$$h_t = A_t h_{t-1} + B_t x_t$$ $$y_t = C_t h_t$$

If we unroll this recurrence from the first token to the last, the final output $y_t$ becomes a weighted sum of all prior inputs, scaled by cumulative products of the transition matrix $A$. When $A_t$ is structured simply as a scalar times an identity matrix, this unrolled history can be rewritten exactly as a matrix multiplication.

Remarkably, this matrix possesses a 1-semiseparable structure that matches the exact matrix factorization found in linear attention, where a running key-value matrix is updated over time.

This duality provides a computational "Rosetta Stone." It means a single model can be materialized in two entirely different ways depending on the context:

  1. The Recurrent Form ($O(N)$): Ideal for autoregressive generation, where maintaining a compact, fixed-size hidden state enables fast, efficient inference.
  2. The Materialized Matrix Form ($O(N)$ via block decomposition): Ideal for training on modern hardware (like GPUs). Instead of stepping through the sequence sequentially, the structured matrix can be computed in parallel, unlocking the same training efficiency that propelled Transformers to dominance.

Expanding the Unified Spectrum

Once the connection between structured matrices and attention is established, other seemingly disparate sequence modeling techniques fall into place along the same spectrum.

Implicit Attention and Gated RNNs

Models like RWKV and Griffin avoid explicit attention matrices to maintain sub-quadratic efficiency. However, Itamar Zimerman et al. demonstrated that these gated-linear RNNs are not actually attention-free. By reframing their recurrence relations as data-controlled linear operators, the authors derived a unified "implicit attention" matrix inherent within these models, extending explainability tools originally designed for Transformers to recurrent architectures.

Test-Time Training as Linear Attention

Test-Time Training (TTT) layers represent a different philosophy: they treat the hidden state of a sequence model as a machine learning model itself, updating its weights via gradient descent during the forward pass to memorize context. Yet, Junchen Liu et al. proved that TTT with key-value binding is secretly mathematically equivalent to a learned linear attention operator. This reframing allowed the authors to bypass the slow sequential weight updates during training, unlocking a parallel formulation that achieved a 4x speedup in inference throughput.

Convolutions and Recurrence

The unification also encompasses convolutional networks. Shaojie Bai et al. showed that truncated RNNs can be mathematically framed as a special case of temporal convolutions with tied weights. This demonstrates that recurrence and convolution are not competing paradigms, but rather distinct points on a single structural spectrum of hidden state management.

Why the Unified View Matters

Understanding the underlying mathematical unity of sequence models shifts the research focus away from choosing a specific architecture and toward optimizing the properties of the shared underlying operators.

From a hardware perspective, it allows optimizations developed for one model class to instantly benefit others, as seen in how Mamba-2 leverages GPU matrix-multiplication units via its dual linear-attention form. From an expressivity standpoint, modeling these architectures through unified frameworks—such as the multivariate polynomial approach used by Edo Cohen-Karlik et al.—allows researchers to rigorously prove why selective SSMs can match Transformer performance while maintaining robust generalization bounds over long sequences.

Ultimately, this unification bridges machine learning with cognitive science. As explored by Kazuki Irie et al., framing modern linear Transformers and SSMs as Fast Weight Programmers connects high-performance AI architectures directly to biologically plausible models of synaptic plasticity in the brain. Just as neuroscientists like V. S. Chandrasekhar Pammi et al. and Krishna Prasad Miyapuram et al. have long sought a unified theory for how the human brain perceives, learns, and executes sequences across different cognitive domains, computer science is converging on a single, elegant truth: sequence modeling is fundamentally an exercise in manipulating structured linear operators.

Key Foundational Work

Investigation of sequence processing: A cognitive and computational neuroscience perspective (2005) — contributed 2 concept elements to this direction.
A Multi-disciplinary Approach to the Investigation of Aspects of Serial Order in Cognition * (2002) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_sequence_modeling_framework_c9ea3c73,
  title = {unified sequence modeling framework},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-sequence-modeling-framework/rdr_c9ea3c73eda74b8c8ee3d804b1ba939f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_sequence_modeling_framework_c9ea3c73_v1,
  title = {unified sequence modeling framework},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-sequence-modeling-framework/rdr_c9ea3c73eda74b8c8ee3d804b1ba939f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake