Streaming Attention for Real-Time Sequence Models

44 concepts42 papers1 proposal

Standard sequence-to-sequence models are inherently "offline." When computing standard global attention, the model must calculate a softmax over the entire input sequence to determine the relevance of each token. This mathematical dependency means that to generate the very first token of an output, the model must already have access to the final token of the input.

For real-time applications like live speech transcription, interactive voice assistants, or simultaneous translation, this global lookahead requirement creates an unacceptable latency bottleneck. As noted by Chung-Cheng Chiu et al. in State-of-the-art Speech Recognition With Sequence-to-Sequence Models, forcing a system to wait for an entire utterance to finish before beginning transcription makes real-time interaction impossible.

To solve this, a prominent line of research focuses on breaking global sequence dependencies, transforming offline attention mechanisms into streaming, online operators that process data incrementally as it arrives.


Monotonic Attention: Replacing Global Softmax with Halting Decisions

The core intuition behind streaming attention is that many real-world sequence tasks possess a naturally progressive, left-to-right alignment. In speech recognition, for instance, the order of spoken words strictly mirrors the order of the audio signal.

To exploit this property, Colin Raffel et al. introduced a paradigm shift in Online and Linear-Time Attention by Enforcing Monotonic Alignments. Instead of computing a soft probability distribution over the entire sequence, the model inspects input frames sequentially. At each frame, it makes a hard binary decision: either "halt" and emit an output token based on the current frame, or move forward to the next frame. Because the model never looks back or looks ahead globally, it operates in linear time and can generate outputs instantly as data streams in.

However, forcing the model to focus on a single, exact frame can severely degrade performance; speech features are often distributed across time. To resolve this, Chung-Cheng Chiu et al. proposed Monotonic Chunkwise Attention (MoChA) in Monotonic Chunkwise Attention.

Standard Global Attention:
Output Token ──> Softmax over ALL Input Frames [1, 2, 3, ..., T] (Requires full sequence)

Monotonic Chunkwise Attention (MoChA):
Input Stream:  [Frame 1] ──> [Frame 2] ──> [Frame 3] ──> [Frame 4]
                                              │             │
                                        Halt Decision? ──> YES!
                                              │             │
                                              └─► [Softmax Window] ──► Output Token

As illustrated above, once MoChA's binary mechanism decides to halt at a specific frame, it opens a small, fixed-size window over the preceding frames. The model then performs standard soft attention only within this localized chunk. This hybrid approach captures local acoustic context while strictly forbidding future lookahead, successfully preserving streaming capabilities.


Chunk-Wise Processing and Context Inheritance

While monotonic attention works well for recurrent architectures, adapting the highly parallel Transformer architecture for streaming requires a different approach. In a standard Transformer, every token attends to every other token across the entire sequence duration.

To make Transformers causal, researchers divide the continuous input stream into localized, fixed-size blocks. However, as Haoran Miao et al. demonstrate in Transformer-based Online CTC/attention End-to-End Speech Recognition Architecture, naive chunking introduces a severe accuracy penalty. When you abruptly cut a sequence into isolated blocks, the model loses historical context from previous chunks, leading to fragmented and inaccurate generations.

The solution is context inheritance. In Transformer ASR with Contextual Block Processing, Emiru Tsunoo et al. introduced a mechanism where a dedicated context vector summarizes the historical information of a block and passes it forward to the next block.

[Chunk N-1] ───► Attention Processing ───► [Context Vector]
                                                   │ (Inheritance)
                                                   ▼
[Chunk N]   ───► Attention Processing ◄────────────┘

This allows the attention layer in the current chunk to remain aware of past context without needing to recompute or store the raw keys and values of all historical tokens.

Building on this concept, Jian Luo et al. developed the Memory-Self-Attention Transducer in Unidirectional Memory-Self-Attention Transducer for Online Speech Recognition. By pairing a restricted-window self-attention layer with an explicit recurrent memory unit, the authors ensured that historical context propagates efficiently across time steps, allowing chunk-based models to closely match the accuracy of their offline counterparts.


Adaptive Look-Ahead and Transducers

Fixed chunk boundaries assume that information density is uniform across a stream. In reality, language is highly variable; some words or phonemes require more future context to disambiguate than others.

To handle this dynamically, Chengyi Wang et al. introduced a "Scout Network" framework in Low Latency End-to-End Streaming Speech Recognition with a Scout Network. The system deploys a small, computationally lightweight, fully causal network that runs slightly ahead of the main recognition model to predict word boundaries in real time. When the Scout Network detects a boundary, it dynamically adjusts the look-ahead window size for the larger attention model. This ensures that the heavy recognition model only pauses for future frames when absolutely necessary to resolve ambiguity, minimizing overall latency.

Alternatively, the Neural Transducer framework, pioneered by Navdeep Jaitly et al. in An Online Sequence-to-Sequence Model Using Partial Conditioning, completely decouples the strict step-by-step alignment. The transducer processes incoming audio blocks and uses a local RNN decoder to emit a variable number of tokens (including a blank token indicating no output) for each block.

By replacing global sequence dependencies with localized chunking, context inheritance, and adaptive halting, these architectures successfully bridge the gap between the high accuracy of global attention and the strict latency constraints of real-time deployment.

Key Foundational Work

Learning Online Alignments with Continuous Rewards Policy Gradient (2016) — contributed 1 concept elements to this direction.
An Online Sequence-to-Sequence Model Using Partial Conditioning (2016) — contributed 1 concept elements to this direction.
An On-Line Time Warping Algorithm for Tracking Musical Performances (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_global_sequence_dependency_07f99c01,
  title = {global sequence dependency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-sequence-dependency/rdr_07f99c018bbf44059388c9422f3a5bc3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_global_sequence_dependency_07f99c01_v1,
  title = {global sequence dependency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-sequence-dependency/rdr_07f99c018bbf44059388c9422f3a5bc3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake