In traditional sequence-to-sequence models for speech and audio processing—whether converting text to speech (TTS), transcribing audio (ASR), or translating spoken language (S2ST)—generation has historically been autoregressive. An autoregressive model predicts an output sequence token-by-token or frame-by-frame, where each step depends on the outputs generated in all preceding steps.
While highly accurate, this sequential dependency imposes a severe computational bottleneck: generating a sequence of length $N$ requires $N$ sequential forward passes through the network. For high-fidelity audio or long sentences, this $O(N)$ latency makes real-time deployment exceptionally difficult.
Non-autoregressive (NAR) or parallel architectures break this bottleneck. By decoupling the generation of each token or frame from its neighbors, NAR models can synthesize entire sequences simultaneously in a single forward pass ($O(1)$ complexity) or a small, fixed number of refinement steps. This shift unlocks massive inference speedups—often ranging from 10x to over 900x faster than real-time—fundamentally changing how speech models are deployed on modern parallel hardware.
The Core Challenge: The One-to-Many Mapping Problem
If parallel generation is so efficient, why did the field rely on sequential models for so long? The primary obstacle is the one-to-many mapping problem (also known as the acoustic multimodality problem).
In speech synthesis or translation, there are many valid ways to pronounce a sentence or translate a phrase. An autoregressive model handles this easily because its step-by-step choices condition future tokens, ensuring the output follows a single, coherent path.
If a naive model attempts to predict all frames simultaneously and independently, it suffers from a coordination failure. For instance, frame 5 might assume a fast speaking rate while frame 6 assumes a slow one. When their independent predictions are combined, the result is a garbled, blurry average of multiple valid paths—leading to gibberish text or unnatural, robotic audio.
To make parallel generation work, researchers have designed structural mechanisms to inject coordination and alignment without restoring the sequential bottleneck.
Overcoming the Bottleneck in Text-to-Speech
In text-to-speech synthesis, the primary challenge is mapping a short sequence of text phonemes to a much longer sequence of acoustic frames (such as mel-spectrograms). Early parallel systems relied on knowledge distillation from sequential models. Peng et al. introduced ParaNet, a fully convolutional non-autoregressive TTS model that distills attention maps from an autoregressive teacher to generate mel-spectrograms in parallel, achieving a 46.7x speedup over sequential baselines (Non-Autoregressive Neural Text-to-Speech).
To completely eliminate the reliance on unstable attention mechanisms, modern NAR architectures explicitly predict prosodic features like duration and pitch to anchor the parallel generation process:
- Explicit Prosody Predictors: Łańcucki developed FastPitch, which uses a feed-forward Transformer to explicitly predict the duration and fundamental frequency (pitch) of each phoneme. By conditioning the parallel frame generator on these explicit targets, the model synthesizes high-quality mel-spectrograms over 900x faster than real-time (FastPitch: Parallel Text-to-speech with Pitch Prediction).
- Latent Variable Modeling: Zen et al. leveraged a Variational Autoencoder (VAE) in Parallel Tacotron to capture the continuous, non-linguistic variations of speech. This architectural choice provides a global coordination signal that eliminates the word-skipping and repeating errors common in autoregressive attention layers (Parallel Tacotron: Non-Autoregressive and Controllable TTS).
- Hard Alignment Optimization: Rather than relying on external alignment teachers, Wang et al. proposed AlignTTS, which uses an alignment loss derived from the classical Baum-Welch algorithm to directly learn the text-to-frame mapping within a feed-forward structure (AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment).
Parallel Decoding in Speech Recognition and Translation
For automatic speech recognition (ASR) and speech translation (ST), the problem is inverted: the model must map a long, continuous audio signal to a shorter sequence of discrete text tokens.
Connectionist Temporal Classification (CTC) serves as a powerful alignment prior for parallel decoding because it enforces a monotonic mapping between audio frames and text tokens. However, standard CTC assumes conditional independence between output tokens. To recover linguistic coherence without sacrificing speed, researchers combine CTC with iterative refinement:
- Masked Language Modeling: Watanabe et al. introduced Mask CTC, which uses a CTC module to generate an initial text sequence in a single pass. It then identifies low-confidence tokens, masks them, and uses a masked language modeling decoder to "fill in the blanks" in parallel, yielding up to a 100x speedup over traditional autoregressive beam search (Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict).
- Single-Pass Architectures: Yi et al. developed LASO (Listen Attentively, and Spell Once), which completely eliminates sequential token dependencies by using a feed-forward architecture to generate entire sentences in a single forward pass (Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition).
- Hybrid Decoders: For tasks requiring complex word reordering like speech translation, Inaguma et al. proposed Orthros, which pairs a fast non-autoregressive decoder with an autoregressive rescorer, bypassing the sequential bottleneck while preserving translation quality (Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder).
- Industrial Scaling: Yan et al. demonstrated that parallel ASR can scale to massive industrial datasets using Paraformer, which utilizes a Continuous Integrate-and-Fire predictor alongside a "glancing" language model sampler to achieve state-of-the-art accuracy with a 10x speedup (Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition).
Iterative Parallel Refinement: Flows and Diffusion
The latest paradigm in non-autoregressive audio generation moves away from strict single-shot generation toward iterative parallel decoding. Instead of executing $N$ steps for $N$ tokens, these models refine the entire sequence simultaneously over a small, fixed number of iterations ($K \ll N$), leveraging continuous normalizing flows, diffusion models, or masked discrete audio codecs.
Autoregressive Generation (O(N)):
[Token 1] ──> [Token 2] ──> [Token 3] ──> ... ──> [Token N]
Iterative Parallel Generation (O(K)):
[Step 1] [Noise, Noise, Noise, ..., Noise] (Predict all simultaneously)
│ │ │ │
[Step 2] [ Token 1, Noise, Token 3, ..., Noise ] (Refine based on confidence)
│ │ │ │
[Step K] [ Token 1, Token 2, Token 3, ..., Token N] (Final high-fidelity output)
- Flow Matching: Le et al. introduced Voicebox, framing speech generation as a text-guided audio infilling task. By training on continuous normalizing flows, Voicebox generates high-fidelity audio in a fixed number of steps, allowing the model to utilize global context from both directions and run 20x faster than autoregressive language models (Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale).
- Confidence-Based Codec Sampling: Borsos proposed SoundStorm, which operates on hierarchical discrete audio tokens. SoundStorm uses a parallel, confidence-based decoding scheme to replace the slow autoregressive generation of audio codecs, synthesizing 30 seconds of consistent audio in just 0.5 seconds (SoundStorm: Efficient Parallel Audio Generation).
- Latent Diffusion: Yang et al. developed SimpleSpeech, which pairs a scalar quantization codec with a transformer-based diffusion model. Operating in a simplified scalar latent space allows the model to bypass complex phoneme alignments entirely while accelerating inference (SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models).
- Cross-Modal Generation: This iterative parallel paradigm extends beyond text to visual inputs. Su demonstrated that predicting all audio tokens simultaneously and refining them iteratively based on visual features enables high-fidelity video-to-audio generation with a 17x speedup over autoregressive alternatives (From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation).
Real-Time and Omnimodal Integration
The massive latency reductions unlocked by non-autoregressive decoding have made real-time, zero-shot speech interaction feasible on resource-constrained hardware and within large language models (LLMs).
For edge deployment, Ji et al. developed MobileSpeech, a parallel mask-based decoding framework that treats speech synthesis as a parallel fill-in-the-blank task across multiple codec layers, enabling high-fidelity voice cloning directly on mobile devices (MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech).
In the domain of conversational AI, non-autoregressive decoders are increasingly integrated directly into the internal representations of LLMs to enable fluid, native voice interactions:
- Simultaneous Text-Speech Generation: Fang et al. introduced LLaMA-Omni, an architecture that feeds the internal hidden states of an LLM directly into a non-autoregressive speech decoder. This allows the model to generate text and speech responses simultaneously, eliminating the latency of traditional cascaded systems (LLaMA-Omni: Seamless Speech Interaction with Large Language Models).
- Sub-Second Omnimodal Latency: To achieve the sub-second response times required for natural human-computer interaction, Yang et al. combined non-autoregressive decoding with CTC loss in OpenOmni, enabling near-instantaneous, emotional speech synthesis within large multimodal models (OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis).