For decades, generative audio systems treated sound as a continuous signal. Whether predicting raw waveforms or intermediate frequency-domain representations like Mel-spectrograms, models relied on continuous regression losses. However, regression models optimize for the mathematical mean. When faced with the highly complex, multi-modal distribution of real-world audio—where a single word can be spoken with infinite variations in pitch, emotion, and background noise—regression models tend to hedge their bets, resulting in blurry, muffled, or averaged outputs.
The breakthrough of modern audio language models (ALMs) comes from a fundamental shift: transforming audio generation from a continuous regression problem into a discrete next-token prediction task, identical to natural language processing. As surveyed by Zhengdong Yang et al., converting continuous speech into a sequence of discrete acoustic units allows a Transformer to treat audio as a native vocabulary. This shift unlocks the exact same scaling laws, parallel architectures, and in-context learning capabilities that powered the revolution in text-based large language models.
How Neural Codecs Tokenize Sound
To treat audio like text, we must first build an alphabet. This is the role of a neural audio codec (such as EnCodec or Descript Audio Codec). Unlike traditional audio compression, a neural codec uses a deep encoder-decoder network paired with a quantization layer to map continuous waveforms into discrete codes.
If we were to use standard Vector Quantization (VQ)—where a continuous vector is mapped to the single closest vector in a learned codebook—we would face an impossible trade-off. To capture the fine-grained acoustic details of a 24kHz audio signal, the codebook would need to be astronomically large, making it impossible to train.
To solve this, neural codecs use Residual Vector Quantization (RVQ). Instead of a single codebook, RVQ uses a cascade of multiple codebooks structured hierarchically:
- The continuous latent vector from the audio encoder is quantized by the first codebook.
- The residual error (the difference between the original vector and its quantized representation) is calculated.
- This residual error is then passed to the second codebook for quantization.
- The process repeats across $N$ layers (often 8 to 16 codebooks).
Continuous Audio Latent ──> [ Codebook 1 ] ──> Token 1
│
Residual Error
▼
[ Codebook 2 ] ──> Token 2
│
Residual Error
▼
[ Codebook 3 ] ──> Token 3
This architecture creates a 2D matrix of discrete tokens (Time $\times$ Codebooks) that naturally disentangles the audio signal. The first few codebook layers capture coarse, high-level structural information, such as phonemes, linguistic content, and primary prosody. The deeper layers capture the remaining high-frequency details, such as speaker timbre, background textures, and acoustic environment. As noted by Haibin Wu et al., this quantized latent space bridges the gap between raw acoustic waveforms and symbolic processing architectures.
Sequence Modeling Over Multi-Stream Tokens
While RVQ successfully discretizes audio, it introduces a new challenge for standard language models. Text is a single stream of tokens, but RVQ produces multiple parallel streams of tokens at every time step.
To model this efficiently, the seminal VALL-E framework introduced by Chengyi Wang et al. splits the generation process into a two-stage hierarchical pipeline: an Autoregressive (AR) stage followed by a Non-Autoregressive (NAR) stage.
Text/Prompt ──> [ Autoregressive Transformer ] ──> Codebook 1 Tokens (Sequential)
│
▼
[ Non-Autoregressive Transformer ]
│
▼
Codebooks 2-N Tokens (Parallel)
1. The Autoregressive Stage (Global Structure)
The AR model is responsible for mapping the input conditioning signal (such as text phonemes) to the first layer of the RVQ codebook. Because the first layer contains the core semantic and structural choices of the speech, it can be predicted sequentially, token by token, conditioned on all previously generated tokens. This allows the model to dynamically determine the duration, rhythm, and intonation of the speech. To make this process highly efficient, Théodor Lemerle et al. demonstrated that replacing quadratic-complexity transformers with linear attention blocks can significantly accelerate this sequential prediction stage without sacrificing voice quality.
2. The Non-Autoregressive Stage (Acoustic Texture)
Once the first layer of tokens is generated, the NAR model takes over to fill in the remaining codebook layers. Because the global structure and timing are already locked in by the first layer, the deeper layers do not need to be predicted step-by-step over time. Instead, the NAR model predicts the tokens for layers 2 through $N$ in parallel, conditioned on the first layer's tokens. This hybrid approach provides the best of both worlds: the AR stage ensures global consistency and natural prosody, while the NAR stage provides fast, parallel generation of high-fidelity acoustic details.
Why the Discrete Paradigm Works
Framing audio as a language modeling task fundamentally changes how models interact with sound, yielding several distinct advantages over continuous methods:
In-Context Learning and Zero-Shot Cloning
Because the Transformer treats acoustic tokens exactly like words, it inherits the ability to perform in-context learning. In zero-shot text-to-speech tasks, a short acoustic prompt (e.g., a 3-second clip of an unseen speaker) is simply prefixed to the generation sequence. The model naturally continues the sequence, matching the speaker's identity, emotional tone, and recording environment without any fine-tuning or explicit parameter optimization (Chengyi Wang et al.).
Signal Processing as Translation
Tasks like speech enhancement, restoration, and source separation have traditionally been treated as signal-filtering problems. In the discrete token paradigm, they become translation problems.
When a model like Genhancer (Jiaqi Su et al.) or MaskSR (Xiaoyu Liu et al.) encounters noisy or corrupted audio, it maps those degraded inputs to clean target tokens. Instead of mathematically subtracting noise, the model uses its learned acoustic priors to re-synthesize a clean version of the speech from scratch. As shown by Huaying Xue et al., this generative next-token prediction avoids the muffled, artifact-ridden outputs typical of deterministic regression filters.
Seamless Cross-Modal Alignment
Aligning audio with other modalities becomes straightforward when everything shares a discrete sequence format. For example, FoleyGen (Xinhao Mei et al.) and C3LLM (Zixuan Wang et al.) demonstrate that video-to-audio generation can be framed as a sequence-to-sequence translation task. A transformer decoder can ingest visual features and directly predict synchronized sound-effect tokens, bypassing the need for intermediate spectrogram-to-waveform vocoders and ensuring precise temporal alignment.
Co-Designing the Tokenizer and the Language Model
The initial wave of audio language models treated the neural codec and the language model as independent black boxes, using off-the-shelf compression codecs as tokenizers. However, recent work highlights a fundamental tension between compression efficiency and language model predictability.
Traditional audio codecs use overlapping frames and look-ahead mechanisms to maximize reconstruction fidelity at low bitrates. But as Shih-Lun Wu et al. discovered, this introduces severe information leakage across time steps. When a codec frame contains information about the future, it violates the causal assumptions of an autoregressive language model, making the tokens significantly harder to predict.
By co-designing these components—specifically by forcing the codec to use strictly non-overlapping encoder frames and introducing "codebook dropout" during training—the authors demonstrated that one can slightly sacrifice raw compression fidelity to drastically improve token predictability. This co-design paradigm doubles inference speed while simultaneously improving the clarity, naturalness, and intelligibility of the generated audio, marking the next frontier in unified audio-text intelligence.
Key Foundational Work
FoleyGen: Visually-Guided Audio Generation (2023) — contributed 3 concept elements to this direction.
Bahasa Harmony: A Comprehensive Dataset for Bahasa Text-to-Speech Synthesis with Discrete Codec Modeling of EnGen-TTS (2024) — contributed 2 concept elements to this direction.
Text-to-Song: Towards Controllable Music Generation Incorporating Vocal and Accompaniment (2024) — contributed 2 concept elements to this direction.