Discrete Speech Tokens for Generative Language Modeling

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2311 papers366 conceptsfrozen 2026-10-04

Traditional speech systems are built like a relay race: an Automatic Speech Recognition (ASR) model transcribes audio to text, a Large Language Model (LLM) processes that text, and a Text-to-Speech (TTS) model synthesizes a response. This "cascaded" approach is functional but fragile. It loses the rich nuances of prosody, struggles with unwritten languages, and suffers from error propagation at every handoff.

A new direction in research seeks to eliminate these boundaries by treating speech not as a continuous wave to be transcribed, but as a sequence of discrete tokens to be "read" directly by a language model. By discretizing audio into a symbolic vocabulary, researchers like Dong Zhang and Shimin Li in SpeechGPT have shown that LLMs can perceive and generate speech natively, using the same next-token prediction objective that powers GPT-4.

The Discretization Bridge

The fundamental challenge is that audio is continuous and high-dimensional, while LLMs are designed for discrete symbols. To bridge this gap, researchers use self-supervised learning (SSL) models—such as HuBERT or WavLM—to extract high-level features from raw audio. These features are then mapped to a finite set of "speech units" using clustering algorithms like k-means.

As Alexei Baevski and Michael Auli demonstrated with vq-wav2vec, this process effectively turns a sound wave into a "pseudo-language." Once audio is represented as a string of integers (e.g., unit 42, unit 109, unit 42), it can be added to an LLM’s vocabulary just like words in English or Mandarin. This allows the model to leverage its pre-trained linguistic reasoning for audio tasks, a concept Frank Palma Gomez and colleagues used to turn LLMs into cross-modal retrieval systems.

Semantic vs. Acoustic Units

Not all speech tokens are created equal. Research has converged on a hierarchy of discretization:

  1. Semantic Units: Derived from the deeper layers of SSL models, these tokens capture phonetic and linguistic content while ignoring the speaker's identity or background noise. Ella Zeldes and Yossi Adi found that using these units in LOTHM significantly improves the stability of speech synthesis for languages like Hebrew.
  2. Acoustic Units: These capture the "how" of speech—pitch, energy, and speaker style. In PolyVoice, Lu Lu and the ByteDance team used a combination of semantic and acoustic tokens to perform speech-to-speech translation that preserves the original speaker's voice across languages.

By disentangling these streams, as explored by Adam Polyak in Speech Resynthesis from Discrete Disentangled Representations, models can perform "voice conversion" by simply swapping the acoustic tokens while keeping the semantic tokens constant.

Textless Modeling and In-Context Learning

One of the most provocative implications of this research is "textless" NLP. If we can represent speech as tokens, we no longer need text as an intermediate medium. This is a breakthrough for the thousands of unwritten languages globally. Chen Zhang and Xu Tan demonstrated this with UWSpeech, which translates speech between languages without ever seeing a single written word.

Furthermore, these tokens allow speech models to inherit the "emergent" properties of LLMs. Tu Anh Nguyen and the Meta AI team showed in dGSLM that models can learn the messy dynamics of human conversation—like laughter and turn-taking—directly from raw audio. Similarly, Ming-Hao Hsu and Hung-yi Lee proved that speech models can perform In-Context Learning, adapting to new tasks like emotion recognition simply by being "prompted" with a few audio examples.

Efficiency and the Path Forward

While continuous features still hold a slight edge in raw accuracy for some tasks, discrete tokens offer massive advantages in efficiency. Xuankai Chang and Brian Yan showed that discretized inputs can lead to a 100x reduction in storage and a 2x speedup in training.

The field is now moving toward more sophisticated tokenizers. Instead of simple k-means, Zhichao Huang introduced RepCodec, a neural representation codec that learns to reconstruct high-level features, creating tokens that are even more semantically rich. As these tokenization methods mature, the distinction between "language models" and "speech models" is likely to vanish, resulting in a single, unified architecture that hears, thinks, and speaks in a shared symbolic space.

Papers in this version

Showing 60 of 311 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 286 papers
  2. Version 1 · 2026-10-04 · 311 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_self_supervised_speech_discretization_04813a4f,
  title = {self-supervised speech discretization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-supervised-speech-discretization/rdr_04813a4fa25646358c1fc0db1db1665a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_discrete_speech_tokens_for_generative_la_04813a4f_v1,
  title = {Discrete Speech Tokens for Generative Language Modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-speech-tokens-for-generative-language-modeling/rdr_04813a4fa25646358c1fc0db1db1665a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}