Non-Autoregressive Decoding for Real-Time Speech Recognition

19 concepts19 papers1 proposal

Standard automatic speech recognition (ASR) systems rely on autoregressive decoding. To translate an audio signal into text, an autoregressive model generates text tokens sequentially, character by character or word by word. Each new token requires a complete forward pass through the network because its prediction depends directly on all previously generated tokens.

While highly accurate, this sequential dependency creates a severe computational bottleneck. If an audio clip maps to a 50-token sentence, the decoder must run 50 times in series. This linear time complexity ($O(N)$ steps) prevents the system from fully utilizing the parallel processing capabilities of modern hardware like GPUs and TPUs, leading to high inference latency that hampers real-time applications such as live captioning and voice assistants.

To break this bottleneck, researchers have developed Non-Autoregressive (NAR) Transformers. These models aim to generate the entire text sequence simultaneously in a single forward pass ($O(1)$ steps), unlocking massive parallelization and accelerating inference speeds by up to 50x.


The Core Challenge: Conditional Independence

If parallel generation is so efficient, why did we use autoregressive models in the first place? The difficulty lies in the multimodality problem and the conditional independence assumption.

When a model predicts all output tokens simultaneously without knowing what the adjacent tokens are, it treats each position as independent. For example, if an audio segment can be transcribed as either "thank you" or "thanks," an independent parallel predictor might output a garbled mixture like "thanks you" or repeat tokens because the positions cannot coordinate with each other. Furthermore, because the model predicts everything at once, it must somehow determine the exact length of the target text sequence before generating it.

To make non-autoregressive speech recognition work, architectures must find alternative mechanisms to coordinate token dependencies and predict sequence lengths accurately without falling back on step-by-step generation.


Mechanism 1: Acoustic Anchors and Length Prediction via CTC

One of the most effective ways to solve the length and alignment problem is to use Connectionist Temporal Classification (CTC) as an acoustic anchor. CTC maps acoustic frames directly to token labels or "blank" tokens, providing a natural framework for alignment without sequential constraints.

Audio Frames:  [ F1 ] [ F2 ] [ F3 ] [ F4 ] [ F5 ] [ F6 ]
                 |      |      |      |      |      |
CTC Output:    [ c ]  [ _ ]  [ a ]  [ a ]  [ _ ]  [ t ]
                 \             /                    /
Spike Trigger:

*           *                    *  --> Length = 3

Jiangyan Yi et al. leveraged this in Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition to eliminate sequential dependencies entirely. To refine this process, Jiangyan Yi et al. also developed the Spike-Triggered Non-Autoregressive Transformer (Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition), which uses the CTC module to identify acoustic "spikes." These spikes precisely predict the target sentence length and trigger the decoder only at relevant acoustic boundaries.

Similarly, Haoneng Luo et al. demonstrated that using CTC to anchor acoustic boundaries, combined with a context-aware decoder, allows the model to capture linguistic patterns without sacrificing parallel execution (Boundary and Context Aware Training for CIF-based Non-Autoregressive End-to-end ASR).


Mechanism 2: Continuous Integrate-and-Fire and Glancing Samplers

Instead of relying solely on hard frame-level CTC selections, some architectures use a Continuous Integrate-and-Fire (CIF) mechanism to dynamically calculate token lengths. CIF integrates acoustic information across time frames until a specific threshold is reached, signaling that a token boundary has been found and "firing" a consolidated acoustic representation to the decoder.

Acoustic Weights:  [0.3] -> [0.4] -> [0.5]  --> Sum = 1.2 (>= 1.0)
                               |
                         [ FIRE TOKEN ] 

This approach is central to Paraformer, an architecture proposed by Ian McLoughlin 0001 et al. in Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition. Paraformer combines a CIF predictor with a "glancing" language model sampler during training. The glancing sampler randomly substitutes some target tokens into the decoder input, forcing the model to learn bidirectional token dependencies. This allows the system to generate highly accurate text in a single step, making it robust enough for industrial deployment toolkits like FunASR (Zerui Li et al. in FunASR: A Fundamental End-to-End Speech Recognition Toolkit).


Mechanism 3: Masked Language Modeling and Denoising

Another intuition for bypassing the sequential bottleneck is to treat speech recognition as a fill-in-the-blanks or denoising task rather than a left-to-right generation task.

Nanxin Chen et al. introduced the Audio-Factorized Masked Language Model (A-FMLM) in Non-Autoregressive Transformer for Speech Recognition. Instead of predicting tokens one by one, the model accepts a completely masked token sequence and uses bidirectional context to fill in all the blanks simultaneously based on the audio features.

To handle the residual errors caused by independent token choices, Nanxin Chen et al. also proposed the Align-Denoise framework (Align-Denoise: Single-Pass Non-Autoregressive Speech Recognition). This framework trains the decoder to take a noisy, independently predicted alignment distribution and "denoise" it into a clean, coherent text sequence in a single pass.


Mechanism 4: Bidirectional Decoders and Latent Variables

Autoregressive decoders are strictly causal; they can only look at the past. Non-autoregressive decoders, however, can look at both past and future context simultaneously because the entire sequence template is available at once.

Autoregressive (Causal):      [Token 1] ---> [Token 2] ---> [Token 3] ---> [Token 4]

Non-Autoregressive (Bi-di):   [Token 1] <---> [Token 2] <---> [Token 3] <---> [Token 4]

The authors of Non-autoregressive Transformer with Unified Bidirectional Decoder for Automatic Speech Recognition (Chuan-Fei Zhang et al.) utilized a Unified Bidirectional Decoder to capture global context across the entire phrase, matching the accuracy of traditional models while running nearly 50 times faster.

To explicitly model token correlations without sequential steps, Shinji Watanabe et al. introduced LV-CTC (LV-CTC: Non-autoregressive ASR with CTC and latent variable models). By embedding a continuous latent space within the CTC framework, the authors provided a mathematical channel for tokens to share global structural information, successfully overcoming the conditional independence limitation.


Why the Idea Works: Scaling and Robustness

Non-autoregressive decoding works because it shifts the computational burden away from serial loops and into parallelizable neural operations. This architectural shift yields several distinct advantages:

  1. Maximizing Encoder Efficiency: In speech translation and ASR, optimizations made to speed up the encoder are often bottlenecked by a slow autoregressive decoder (Jinming Zhao et al. in RedApt: An Adaptor for wav2vec 2 EncodingFaster and Smaller Speech Translation without Quality Compromise). NAR decoders ensure that the entire pipeline remains highly parallel.
  2. Synergy with Pre-trained Models: NAR models scale exceptionally well when paired with large, self-supervised acoustic and language representations like wav2vec 2.0 and BERT, as shown by Keqi Deng et al. (Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models) and Chung-Cheng Chiu et al. (Pushing the Limits of Non-Autoregressive Speech Recognition).
  3. Streaming Capabilities: By applying blockwise attention and dynamic overlapping decoding, the parallel nature of NAR models can even be adapted to streaming audio inputs, providing ultra-low latency responses (Tianzi Wang et al. in Streaming End-to-End ASR Based on Blockwise Non-Autoregressive Models).
  4. Long-Sequence Robustness: In a comparative study by Yosuke Higuchi et al. (A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation), NAR models demonstrated a unique advantage over autoregressive models: superior robustness when transcribing very long audio sequences. Because NAR models do not feed their own prior predictions back into themselves, they are immune to the compounding error loops that cause autoregressive decoders to fail or hallucinate on long inputs.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_autoregressive_inference_latency_47338940,
  title = {autoregressive inference latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/autoregressive-inference-latency/rdr_47338940b4244096a1b6f808e567efbf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_autoregressive_inference_latency_47338940_v1,
  title = {autoregressive inference latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/autoregressive-inference-latency/rdr_47338940b4244096a1b6f808e567efbf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake