Monotonic Chunkwise Attention for Streaming Sequences

This direction is merged into another direction: streaming attention mechanism. Its last version stays readable and citable here.

Version 1 of 125 papers27 conceptsfrozen 2026-10-04

Standard attention mechanisms are inherently "offline." To calculate the attention weights for a single output token, a model must look at every single input token in the sequence. In tasks like speech recognition or live translation, this creates a fundamental bottleneck: the system cannot begin generating a response until the speaker has finished talking. Furthermore, the computational cost of this "soft attention" grows quadratically with the length of the sequence, making it impractical for long-form, unsegmented audio.

To solve this, researchers have developed Monotonic Chunkwise Attention (MoChA) and its derivatives. These mechanisms allow models to process information linearly and incrementally, enabling real-time performance without sacrificing the accuracy of global attention.

The Problem with Global Soft Attention

In a traditional sequence-to-sequence model, the decoder uses a "soft" attention mechanism to assign a weight to every part of the input. While powerful, this approach has two major flaws for real-time applications:

  1. Latency: The model requires the entire input sequence ($T$) to be available before it can compute the first output token ($U$), leading to $O(TU)$ time complexity.
  2. Memory: Storing the entire input sequence and the resulting attention matrix is memory-intensive, especially for on-device applications like smartwatches or hearing aids.

As Chung-Cheng Chiu and Colin Raffel noted in their seminal work on Monotonic Chunkwise Attention, online attention can match the accuracy of offline models if the input and output sequences are naturally aligned, as they are in speech.

The Monotonic Constraint

The core intuition behind these streaming models is the monotonicity constraint. In speech, the order of spoken words generally matches the order of the transcript. We don't need to look at the end of a sentence to understand the first word.

MoChA implements this by replacing the global softmax with a "hard" decision process. At each step, the model decides whether to "stay" at the current input frame or "emit" an output and move forward. This decision is modeled as a Bernoulli random variable. Once the model decides to stop at a specific point in the input, it only looks at that point and its immediate history. This allows the model to generate output tokens as soon as it has seen enough of the input, drastically reducing latency.

From Points to Chunks

While a strictly monotonic "point" attention is efficient, it often lacks the context needed to resolve ambiguities in speech. Chung-Cheng Chiu and Colin Raffel improved this by introducing chunkwise attention.

Instead of attending to just one frame, the model attends to a small, fixed-size "chunk" of frames preceding the chosen point. This provides a local window of context—similar to how a human listener might hold the last few syllables in short-term memory—while still maintaining the ability to process the stream incrementally.

This approach has been successfully deployed in industrial settings. For instance, Kwangyoun Kim and Dhananjaya Gowda at Samsung used MoChA to build an on-device ASR system that fits in 39MB and achieves latencies as low as 140ms.

Blockwise and Causal Architectures

As the field moved from RNNs to Transformers, the "chunking" concept evolved into blockwise or streaming attention masks. In these architectures, the input is divided into fixed temporal increments.

  • Causal Constraints: To prevent the model from "cheating" by looking into the future, researchers like Siqi Ouyang and Lei Li utilize streaming encoders with chunk-wise causal attention. Each block can only attend to itself and a limited history of previous blocks.
  • Stateful Processing: For tasks like speaker separation, Ori Kabeli and Yossi Adi replaced bidirectional layers with unidirectional variants, ensuring the model's output at time $t$ depends only on frames $\leq t$.
  • Non-Autoregressive Streaming: Even non-autoregressive models, which usually generate sequences all at once, have been adapted for streaming. Tianzi Wang and Shinji Watanabe demonstrated a blockwise NAR system that uses a "Mask-CTC" strategy to refine predictions within local segments, achieving a 10x speedup over traditional models.

Handling Unsegmented Audio

A major challenge in real-world streaming is "unbounded" audio—long recordings with varying amounts of silence. Traditional systems rely on Voice Activity Detection (VAD) to segment audio before processing. However, Hirofumi Inaguma and Tatsuya Kawahara showed that MoChA-based systems can handle unsegmented audio natively. Because the monotonic mechanism only triggers an output when it detects relevant acoustic information, it inherently ignores silent regions, allowing for continuous, VAD-free transcription.

By restricting the receptive field to local, sequential chunks, these researchers have transformed attention from a global, offline operator into a local, streaming one—enabling the next generation of real-time AI assistants and translation tools.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_monotonic_chunkwise_attention_for_stream_ccdd361c,
  title = {Monotonic Chunkwise Attention for Streaming Sequences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/monotonic-chunkwise-attention-for-streaming-sequences/rdr_ccdd361cd44541e8a59d773b3dfed0d0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_monotonic_chunkwise_attention_for_stream_ccdd361c_v1,
  title = {Monotonic Chunkwise Attention for Streaming Sequences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/monotonic-chunkwise-attention-for-streaming-sequences/rdr_ccdd361cd44541e8a59d773b3dfed0d0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}