Parallel Sequence Processing with Temporal Convolutions

231 concepts158 papers2 proposals

For years, sequence modeling was synonymous with Recurrent Neural Networks (RNNs) like LSTMs and GRUs. The underlying assumption was intuitive: because sequences unfold over time, a model must process them step-by-step, maintaining an internal hidden state that updates sequentially.

However, this sequential nature creates a massive computational bottleneck. Modern hardware accelerators (GPUs) thrive on massive parallelism, yet an RNN forces them to wait for step $t-1$ to finish before computing step $t$.

To unlock hardware acceleration, researchers questioned this traditional wisdom. What if we treat the temporal dimension not as a loop, but as a spatial dimension? By applying 1D convolutions across the timeline, we can process all time steps simultaneously. This paradigm shift—often formalized as Temporal Convolutional Networks (TCNs) or fully convolutional sequence modeling—demonstrates that pure convolutional networks can match or exceed the accuracy of recurrent models while training orders of magnitude faster.

The Core Mechanisms: Causality and Dilation

To replace an RNN with a Convolutional Neural Network (CNN), a model must solve two fundamental challenges: it cannot leak information from the future into the past, and it must be able to capture long-range dependencies.

Causal Convolutions

A standard 1D convolution looks both left and right of its center pixel. For sequence tasks like language modeling, audio generation, or real-time forecasting, a prediction at time $t$ can only depend on inputs from time $\le t$.

To enforce this, fully convolutional sequence models use causal convolutions. This is achieved by shifting the convolutional filter or padding the input sequence exclusively on the left. As a result, the output at time step $t$ is mathematically guaranteed to be computed only from elements from the current and previous layers at time $t$ and earlier.

Dilated Convolutions

A simple stack of causal convolutions expands its receptive field linearly with depth. If your filter size is $k$ and you have $L$ layers, the network can only look back $L \times (k-1)$ steps. Capturing a context window of thousands of audio frames or text tokens would require an impractically deep network.

To expand the receptive field exponentially without losing resolution or adding too many parameters, models employ dilated convolutions. A dilated convolution applies a filter over an area larger than its size by skipping input values with a specific step size, known as the dilation factor $d$.

$$\text{Standard Causal } (d=1): \quad [x_{t-2}, x_{t-1}, x_t]$$ $$\text{Dilated Causal } (d=2): \quad [x_{t-4}, x_{t-2}, x_t]$$

By doubling the dilation factor exponentially with network depth ($d = 1, 2, 4, 8, \dots$), the top layer's receptive field grows exponentially. This allows the network to capture long-term temporal dependencies efficiently while ensuring that every single input token within that window contributes to the final representation.

Why It Works: Parallelism and Gradient Stability

The shift from recurrence to convolution provides two massive advantages that explain why these architectures outperform traditional sequential networks in practice.

Massive Parallelism

In an RNN, training a sequence of length $T$ requires $T$ sequential forward steps and $T$ sequential backward steps. The computational complexity is bound to the timeline.

In a fully convolutional architecture, the entire sequence is processed as a single, massive matrix operation. Because the convolutional filters are fixed and independent of the sequence index, the same weights are applied across the entire timeline simultaneously. As highlighted by Shaojie Bai et al. in Convolutional Sequence Modeling Revisited, this structural shift eliminates the sequential bottleneck entirely, maximizing GPU throughput during both training and inference.

Stable Gradient Flow

RNNs notoriously suffer from vanishing and exploding gradients because the backpropagation path scales with the sequence length $T$. Gradients must flow back through an unrolled chain of matrix multiplications across time.

In contrast, a hierarchical convolutional structure provides a fixed-length path for gradients. As Jonas Gehring et al. demonstrate in Convolutional Sequence to Sequence Learning, the maximum distance between any two distant tokens in a dilated convolutional network scales logarithmically, not linearly. This structural property stabilizes training, accelerates convergence, and prevents the information degradation typical of long recurrent chains.

Empirical Success Across Domains

By treating temporal sequences as parallelizable spatial dimensions, fully convolutional architectures have unlocked massive speedups and performance gains across diverse domains:

Key Foundational Work

Fast and Accurate Sequence Labeling with Iterated Dilated Convolutions (2017) — contributed 3 concept elements to this direction.
Deep Voice 3: 2000-Speaker Neural Text-to-Speech (2017) — contributed 2 concept elements to this direction.
Temporal Convolutional Networks for Action Segmentation and Detection (2016) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 159 papers
  2. Version 1 · 2026-10-04 · 209 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_convolutional_sequence_modeling_eaa0bbda,
  title = {convolutional sequence modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/convolutional-sequence-modeling/rdr_eaa0bbda88a445db9d8dacb59c67761d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_convolutional_sequence_modeling_eaa0bbda_v2,
  title = {convolutional sequence modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/convolutional-sequence-modeling/rdr_eaa0bbda88a445db9d8dacb59c67761d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake