Parallel Sequence Modeling without Recurrence

This direction is merged into another direction: sequential processing bottleneck (sequential computation bottleneck). Its last version stays readable and citable here.

Version 1 of 128 papers28 conceptsfrozen 2026-10-04

For years, the Recurrent Neural Network (RNN) was the undisputed engine of sequence modeling. By maintaining a hidden state that evolves step-by-step, RNNs like LSTMs and GRUs could theoretically remember the past to predict the future. However, this sequential nature—where step $t$ cannot be computed until step $t-1$ is finished—has become a fundamental bottleneck.

As hardware has shifted toward massive parallelization, the "one-step-at-a-time" requirement of recurrence prevents models from fully utilizing modern GPUs and TPUs. A growing body of research is now focused on replacing recurrence with architectures that can process entire sequences simultaneously, achieving massive speedups in training and inference without sacrificing accuracy.

The Sequential Bottleneck

The core problem with recurrence is its mathematical dependency: $h_t = f(h_{t-1}, x_t)$. This formula is a wall for parallelization. In tasks like speech recognition or video analysis, where sequences can be thousands of steps long, this dependency leads to significant training latency. Clarence Chen and Zachary A. Pardos note that this bottleneck limits how quickly researchers can iterate on models.

Even when recurrence is used for small sub-tasks, like pooling in a CNN, it introduces a slowdown. Shuai Li found that replacing fixed pooling with recurrent units made training two to three times slower, highlighting that even "smart" recurrence often costs more in time than it gains in performance.

Temporal Convolutions: Parallelism through Depth

One of the most successful alternatives to recurrence is the Temporal Convolutional Network (TCN). Instead of passing a state forward through time, TCNs use a stack of causal, dilated convolutions to look back at the past. Because convolutions can be computed for all time steps at once, they are inherently parallel.

In speech synthesis, Wei Ping and colleagues developed Deep Voice 3, a fully convolutional system that trains an order of magnitude faster than previous recurrent models. Similarly, Mohammad Pezeshki demonstrated in Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks that deep CNNs could outperform LSTMs while training 2.5x faster.

The intuition here is simple: if you want to see further back in time, you don't need a persistent memory; you just need a deeper stack of convolutions or larger "dilation" rates that skip over steps to expand the receptive field. This approach has proven vital for real-time applications. For instance, Tobias Czempiel used causal convolutions in the TeCNO model to recognize surgical phases in real-time, where the latency of an LSTM would be unacceptable.

Attention as a Global Shortcut

While convolutions look at a local window of the past, self-attention mechanisms allow every element in a sequence to "attend" to every other element regardless of distance. This eliminates the need for a hidden state to carry information across long gaps.

In scene text recognition, models like NRTR by Fenfen Sheng and MASTER by Rong Xiao have completely ditched RNNs. By using pure Transformer architectures, these models avoid "attention drift" and train up to 8 times faster than their recurrent predecessors.

However, standard self-attention has its own scaling issues (quadratic complexity). To solve this, researchers are developing "linear" attention and "fast recurrence" hybrids. Tao Lei and the team behind SRU++ use a highly parallelized recurrence that unrolls efficiently on hardware, allowing them to process audio sequences longer than 30 seconds—a feat that traditional Transformers and RNNs struggle to achieve efficiently.

Efficiency Beyond Speed

The move away from recurrence isn't just about training faster; it's about building models that can scale. Weizhe Hua and Quoc V Le introduced FLASH, which uses a Gated Attention Unit to achieve a 12x speedup on long sequences. In reinforcement learning, Ryan Kortvelesy showed that Fast and Forgetful Memory (FFM) can train 100x faster than RNNs by explicitly modeling forgetting in a parallelizable way.

By reframing sequence modeling as a parallel computation—whether through dilated convolutions, global attention, or linearized state updates—researchers are overcoming the sequential bottleneck. The result is a new generation of models that are not only more accurate but are also capable of handling the massive, high-resolution temporal data of the modern world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parallel_sequence_modeling_without_recur_738da55e,
  title = {Parallel Sequence Modeling without Recurrence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallel-sequence-modeling-without-recurrence/rdr_738da55e774f42538e82104d54417fae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_parallel_sequence_modeling_without_recur_738da55e_v1,
  title = {Parallel Sequence Modeling without Recurrence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallel-sequence-modeling-without-recurrence/rdr_738da55e774f42538e82104d54417fae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}