Audio Generation as a Discrete Sequence Modeling Problem

This direction is split into other directions: discrete audio tokenization. Its last version stays readable and citable here.

Version 1 of 180 papers100 conceptsfrozen 2026-10-04

For decades, digital audio processing relied on continuous representations like Mel-spectrograms. While effective for compression, these frequency-domain snapshots created a "regression problem" for generative models. When a model predicts a continuous value and misses, the result is often a blurry, muffled, or "averaged" sound.

A new paradigm, pioneered by researchers like Shinji Watanabe in DiscreTalk, shifts audio generation from signal regression to discrete sequence modeling. By treating sound as a series of "audio words" or tokens, we can apply the same Transformer architectures and scaling laws that revolutionized natural language processing to the world of sound.

The Neural Codec as a Tokenizer

The core mechanism of this shift is the neural audio codec. Unlike traditional codecs (like MP3) designed for human ears, neural codecs like EnCodec or DAC use a Vector Quantized Variational Autoencoder (VQ-VAE) to map raw waveforms into a discrete latent space.

As Felix Kreuk and colleagues demonstrate in AudioGen, these codecs use Residual Vector Quantization (RVQ) to represent a single moment of audio as a hierarchy of tokens. The first layer captures the broad semantic content, while subsequent layers add fine-grained acoustic detail. This discretization allows a model to "choose" a distinct acoustic mode rather than averaging between them, which Jiaqi Su notes in Genhancer is the key to avoiding the muffled outputs of previous generations.

Why Discretization Works: The Power of Next-Token Prediction

Once audio is tokenized, tasks like Text-to-Speech (TTS) or Voice Conversion become translation problems. In Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers, Chengyi Wang and Lei He introduced VALL-E, which treats TTS as a conditional language modeling task.

This approach offers three fundamental advantages:

  1. In-Context Learning: By providing a 3-second "acoustic prompt" (a sequence of tokens), the model can continue the sequence in the same voice, emotion, and environment without explicit fine-tuning.
  2. Scaling Laws: Potsawee Manakul and the team behind SODA have begun mapping the scaling laws for audio, finding that as data and compute grow, these models gain emergent capabilities similar to text-based LLMs.
  3. Robustness: Because the model predicts discrete units, it is less sensitive to low-level signal noise. Krishna C Puvvada found that these tokens can even replace Mel-spectrograms for recognition tasks, offering 20x compression with almost no loss in accuracy.

Unified Architectures for Multi-Modal Tasks

The most significant implication of this research is the move toward "universal" audio models. If audio is just a sequence of tokens, then speech recognition, synthesis, and translation can all live in the same Transformer backbone.

Shujie LIU and the VioLA team demonstrated a single decoder-only model that handles all these tasks by simply changing the input prompt. Similarly, Alexandre Défossez introduced Moshi, a unified speech-text model that achieves 200ms latency—fast enough for natural human-AI dialogue—by processing interleaved streams of audio and text tokens.

This "audio-as-language" framing extends beyond human speech. Tali Dror uses discrete tokens for audio inpainting, allowing models to "imagine" missing musical segments by maintaining long-range harmonic coherence. Even haptic vibrations, as explored by Daniel Hershcovich in HapticLLaMA, can be tokenized using audio codecs to capture the rhythmic nuances of touch.

Challenges in Co-Design

While powerful, the direction faces a "bottleneck" between the codec and the language model. Standard codecs are designed for reconstruction, not predictability. Shih-Lun Wu and Chris Donahue argue in Towards Codec-LM Co-design that we must design codecs specifically for LMs—for example, by using non-overlapping frames to prevent information leakage that makes token prediction "too easy" for the model during training but hard to generalize during inference.

By treating sound as a discrete symbolic language, researchers are moving away from the "black box" of signal processing toward a future where AI can reason about, edit, and generate audio with the same flexibility we currently have with text.

Papers in this version

Showing 60 of 80 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_audio_generation_as_a_discrete_sequence__a68643c1,
  title = {Audio Generation as a Discrete Sequence Modeling Problem},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/audio-generation-as-a-discrete-sequence-modeling-problem/rdr_a68643c1a4a343e595f0a8427aace548},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_audio_generation_as_a_discrete_sequence__a68643c1_v1,
  title = {Audio Generation as a Discrete Sequence Modeling Problem},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/audio-generation-as-a-discrete-sequence-modeling-problem/rdr_a68643c1a4a343e595f0a8427aace548/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}