Replacing Recurrence with Parallel Sequence Modeling

This direction is merged into another direction: sequential processing bottleneck (sequential computation bottleneck). Its last version stays readable and citable here.

Version 1 of 147 papers47 conceptsfrozen 2026-10-04

For years, the standard approach to processing sequences—whether text, audio, or financial data—relied on Recurrent Neural Networks (RNNs). These models process data like a human reading a sentence: one word at a time, updating an internal "memory" at each step. While intuitive, this sequential nature creates a fundamental bottleneck.

Recent research across diverse fields has converged on a shared goal: moving away from step-by-step recurrence toward parallel architectures that can "see" entire sequences at once.

The Sequential Bottleneck

The core problem with recurrence is its linear dependency. To calculate the state of the 100th token, you must first calculate the 99th, which requires the 98th, and so on. As Li Wang and colleagues note in A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization, this prevents training parallelization and often leads to vanishing gradients, where the model "forgets" the beginning of a long sequence by the time it reaches the end.

This limitation isn't just a problem for text. In safety-critical systems like autonomous driving, Yue Zhang explains in Long-term prediction for high-resolution lane-changing data using temporal convolution network that the sequential bottleneck of RNNs hinders the real-time performance necessary for predicting vehicle trajectories.

Parallelizing Time with Convolutions

One way to break the sequential chain is to treat time as a spatial dimension. Temporal Convolutional Networks (TCNs) use layers of filters that slide across the entire sequence simultaneously.

To capture long-range dependencies without needing an impractical number of layers, researchers use dilated convolutions. These filters have "holes" in them, allowing a single layer to look at points far apart in time. Junfu Pu demonstrated this in Dilated Convolutional Network with Iterative Optimization for Continuous Sign Language Recognition, where dilated layers allowed the model to maintain context over long video sequences of sign language without the computational drag of an RNN.

Global Context via Self-Attention

While convolutions are parallel, they are still somewhat local—they only see what falls under their filter. The Transformer architecture solves this by using self-attention, which allows every element in a sequence to interact directly with every other element, regardless of distance.

In DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding, Tao Shen showed that by removing RNN/CNN layers entirely and relying on directional masks, models could capture temporal order more efficiently and with fewer parameters. This shift has proven transformative for audio as well. Tingle Li developed Atss-Net, which uses multi-head attention to isolate a specific speaker's voice from a noisy mixture, achieving state-of-the-art results with half the parameters of previous LSTM-based models.

Capturing Non-Linear Dynamics

Sometimes, time isn't just a line; it's a web of influences. Researchers are increasingly using Graph Convolutional Networks (GCNs) to model these relationships. In Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion, Sijie Mai treats individual time steps as nodes in a graph, allowing the model to dynamically learn connections between distant events. This approach trained up to 7x faster than RNN equivalents by enabling massive parallelization.

Similarly, for financial markets, Anh Tong introduced SigFormer, which combines the mathematical power of path signatures with Transformers to handle the "rough" and irregular patterns of market data that traditional recurrent models struggle to process.

The Efficiency Frontier

The most recent evolution in this direction involves Selective State Space Models like Mamba. These models aim to combine the best of both worlds: the efficient inference of RNNs with the parallel training of Transformers. Yang Cao demonstrated in Mamba4KT that this architecture can break the efficiency-accuracy trade-off in knowledge tracing, delivering faster training and a lower memory footprint than any previous sequential model.

By moving from sequential to parallel processing, these researchers have not only made models faster but have fundamentally improved their ability to understand the complex, long-term dependencies that define our world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_replacing_recurrence_with_parallel_seque_dedb47e6,
  title = {Replacing Recurrence with Parallel Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/replacing-recurrence-with-parallel-sequence-modeling/rdr_dedb47e6d07d41d084072972517916d3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_replacing_recurrence_with_parallel_seque_dedb47e6_v1,
  title = {Replacing Recurrence with Parallel Sequence Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/replacing-recurrence-with-parallel-sequence-modeling/rdr_dedb47e6d07d41d084072972517916d3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}