Generative models for audio have achieved remarkable fidelity, producing speech and music that are often indistinguishable from human recordings. Many of these breakthroughs rely on autoregressive architectures, which model the joint probability of an audio sequence by predicting each sample or discrete token conditioned on all previous ones.
However, this sequential approach introduces a severe computational bottleneck during inference. While a text document might contain a few hundred tokens, raw audio requires tens of thousands of samples per second. Even when compressed into discrete acoustic units using neural audio codecs, a single second of audio still maps to dozens of tokens. Because an autoregressive model must execute a full forward pass for every single token in a sequence, this high token-to-duration ratio results in high inference latency, making real-time streaming, interactive deployment, and large-scale synthesis highly impractical.
To unlock real-time performance, researchers have developed several distinct strategies to bypass or accelerate this sequential generation process.
Fully Parallel Non-Autoregressive Synthesis
The most direct way to eliminate the sequential bottleneck is to generate all audio frames simultaneously in a single forward pass. However, breaking the autoregressive chain introduces a fundamental challenge: alignment. In tasks like text-to-speech (TTS), the input text and output audio frames do not align one-to-one. Autoregressive models handle this dynamically through causal attention mechanisms, but non-autoregressive (NAR) models must establish a fixed temporal grid before generation can occur.
To solve this, parallel models incorporate explicit duration predictors or alignment search algorithms to map input tokens to the time domain. For instance, Łańcucki et al. developed FastPitch, a parallel TTS architecture that utilizes an explicit duration and pitch predictor to lay out phonemes across a parallel time grid, enabling the model to synthesize mel-spectrograms hundreds of times faster than real-time. Rather than relying on an external alignment teacher, Jaehyeon Kim et al. introduced Glow-TTS, which combines a flow-based generative framework with a dynamic programming method called Monotonic Alignment Search. This approach allows the model to internally learn the correct alignment while maintaining a fully parallelized generation pipeline.
Autoregressive (Sequential):
[Token 1] ──> [Token 2] ──> [Token 3] ──> [Token 4] (Requires 4 forward passes)
Non-Autoregressive (Parallel):
[Text Input] ──> [Duration/Alignment Predictor] ──> [Time Grid]
│
┌────────────────┴────────────────┐
▼ ▼ ▼
[Frame 1] [Frame 2] [Frame 3] (Single forward pass)
This parallel philosophy extends to raw waveform synthesis (vocoding). Traditional models like WaveNet, designed by Aäron van den Oord, generate audio sample-by-sample, which requires tens of thousands of sequential steps per second. To overcome this, Ryuichi Yamamoto et al. with Parallel WaveGAN and Kundan Kumar et al. with MelGAN replaced sample-by-sample dependencies with fully convolutional generative adversarial networks (GANs). By leveraging parallel hardware acceleration, these models synthesize entire high-fidelity waveforms in a single step.
Speculative Decoding with Speech Tolerance
An alternative to abandoning autoregressive models entirely is to accelerate them using speculative decoding. This technique uses a small, computationally inexpensive "draft" model to generate a sequence of candidate tokens in parallel, which a larger, high-quality "target" model then verifies or corrects in a single forward pass. If the target model accepts the draft tokens, the system advances by multiple steps at once.
While highly effective in natural language processing, standard speculative decoding fails when applied directly to audio. Speech tokens exhibit massive acoustic variability; multiple distinct token sequences can represent the exact same sound or pronunciation. If the target model strictly checks for exact token matches, it will reject perfectly valid drafts simply because they choose a different, yet acoustically equivalent, token path.
To adapt this mechanism for audio, Bohan Li et al. introduced a specialized framework in Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding. The authors designed a "tolerance mechanism" that relaxes the strict verification criteria. Instead of demanding exact token identity, the target model accepts draft tokens if they fall within an acceptable acoustic similarity threshold. This adjustment dramatically increases draft acceptance rates, delivering a significant speedup without sacrificing synthesis quality. Further expanding on this concept, Tan Dat Nguyen developed a method detailed in Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding, which pairs multi-token prediction with Viterbi-based speculative decoding to validate multiple parallel token streams simultaneously, pushing inference speeds closer to non-autoregressive alternatives.
Iterative Refinement via Flow Matching and Diffusion
Rather than generating audio strictly from left to right or attempting to predict everything in a single pass, iterative non-autoregressive models treat generation as a global refinement process. These architectures initialize the entire output sequence as random noise or masked placeholders and iteratively refine the representation over a fixed number of steps.
Matthew Le et al. applied this paradigm at scale with Voicebox, utilizing a text-guided infilling objective trained via flow matching. Instead of executing thousands of sequential steps, Voicebox treats generation as solving an Ordinary Differential Equation (ODE) that transforms a simple noise distribution into structured acoustic features over a small, deterministic number of refinement steps.
Iterative Refinement (Global Denoising):
Step 0 (Noise): [ ░░░░ ] [ ░░░░ ] [ ░░░░ ] [ ░░░░ ]
│ │ │ │
Step 1 (Coarse): [ █░░█ ] [ ░██░ ] [ █░░░ ] [ ░███ ]
│ │ │ │
Step N (Fidelity): [ Audio1 ] [ Audio2 ] [ Audio3 ] [ Audio4 ]
Similarly, Myeonghun Jeong introduced Diff-TTS, which applies a denoising diffusion framework to mel-spectrogram generation. This approach allows users to dynamically trade audio quality for inference speed by adjusting the number of denoising steps. For discrete token spaces, Ziv et al. demonstrated in Masked Audio Generation using a Single Non-Autoregressive Transformer that a single non-autoregressive transformer can iteratively predict masked audio tokens. By updating multiple masked positions simultaneously during each iteration, the model breaks the strict left-to-right dependency chain and significantly reduces the total number of forward passes required for high-fidelity audio synthesis.
Compressing the Temporal Dimension
A final, orthogonal approach targets the root cause of the sequential bottleneck: the high temporal resolution of audio representations. If an autoregressive model requires fewer tokens to represent a second of audio, the sequence length shrinks, and the number of sequential forward passes decreases proportionally.
Standard neural audio codecs are designed to maximize reconstruction fidelity, which often results in high frame rates that generate dozens of tokens per second. To alleviate this downstream computational burden, Edresson Casanova et al. introduced the Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference. The authors optimized the underlying neural codec to compress speech into a significantly lower frame rate while preserving acoustic clarity. By packing more acoustic information into each discrete token, this method shortens the sequence length across the temporal dimension, directly accelerating both training and real-time inference for autoregressive speech language models.