For years, sequence modeling was defined by a fundamental trade-off. On one hand, Recurrent Neural Networks (RNNs) offered efficient, constant-memory inference but were notoriously slow to train because they processed data one step at a time. On the other, Transformers and Convolutional Neural Networks (CNNs) allowed for massive parallelization during training but suffered from high memory overhead or quadratic computational costs as sequences grew longer.
A new class of architectures has emerged to break this "impossible triangle" by leveraging sequential-parallel duality. These models are designed with a dual mathematical identity: they can be expressed as a parallelizable operation (like a convolution or a matrix multiplication) for high-throughput training and as a recurrent state-space update for efficient, low-latency inference.
The Core Intuition: One Model, Two Views
The breakthrough lies in the realization that certain mathematical operations can be computed in two ways that yield identical results. As Shaojie Bai and J. Zico Kolter demonstrated with Trellis Networks, truncated recurrence is mathematically equivalent to a specific configuration of temporal convolutions.
This duality is most clearly seen in Linear State-Space Models (SSMs). In the Linear State-Space Layer (LSSL), Albert Gu and Tri Dao showed that a sequence modeling layer can be viewed through three interchangeable lenses:
- The Continuous View: A differential equation modeling a dynamical system.
- The Recurrent View: A discrete-time linear recurrence ($h_t = Ah_{t-1} + Bx_t$) for step-by-step inference.
- The Convolutional View: A global convolution ($y = K * x$) that allows the entire sequence to be processed at once during training using Fast Fourier Transforms (FFTs).
By switching between these views, a model like SASHIMI can train on raw audio using parallel convolutions but generate new audio samples one by one with the constant-time efficiency of an RNN.
Bridging Attention and Recurrence
While SSMs emerged from control theory, other researchers arrived at duality by reimagining the Transformer's attention mechanism. The Retentive Network (RetNet), introduced by Yutao Sun and Li Dong, replaces standard attention with a "retention" module. This module uses a dual-form representation that functions as a parallel matrix operation during training and a recurrent state-update during deployment, achieving Transformer-level performance with $O(1)$ inference complexity.
Similarly, the RWKV architecture, developed by Bo Peng and Stella Biderman, unifies the two paradigms into a single framework. It scales to billions of parameters while maintaining the ability to switch between a high-throughput "parallel mode" for GPUs and a memory-efficient "sequential mode" for generation.
The Role of Prefix Scans
The mathematical "glue" that often enables this duality is the parallel prefix scan. In Sequential-Parallel Duality in Prefix Scannable Models, Kazuki Irie and Jacob Andreas explain that if the operations used to combine sequence elements are associative, they can be computed in $O(\log N)$ time using a tree-like parallel structure. This allows models to maintain a recurrent state while still exploiting the massive parallelism of modern hardware.
This principle ensures that the "research-to-production tax"—the need to rewrite models for real-time deployment—is eliminated. As Julian Salazar and colleagues argue in SequenceLayers, enforcing a strict contract where parallel and sequential execution yield identical results allows for automated verification of model behavior across different hardware targets.
Applications Across Domains
The impact of this duality is particularly visible in fields requiring real-time processing of high-resolution data:
- Physiological Monitoring: Kegang Wang and Yuntao Wang developed ME-rPPG, which uses Temporal-Spatial State Space Duality to monitor heart rates on mobile devices with ultra-low latency.
- Reinforcement Learning: Fei Deng introduced S4WM, a world model that uses the convolutional form to train on long trajectories and the recurrent form to "imagine" future states hundreds of steps ahead in real-time.
- Audio and Speech: Models like aTENNuate leverage the duality to perform speech denoising directly on raw waveforms, providing the training stability of spectral methods with the sample-by-sample execution needed for hearing aids.
By treating recurrence and parallelism not as competing paradigms but as dual facets of the same underlying system, this research direction provides a scalable path forward for the next generation of foundation models.