Scaling the Granularity of Discrete Acoustic Units

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 215 papers19 conceptsfrozen 2026-10-04

In the quest to make machines understand and generate speech, researchers have increasingly turned to "discretization"—the process of turning continuous audio waves into a sequence of discrete tokens, much like words in a sentence. While early models relied on a small "alphabet" of sounds, a growing body of research suggests that the key to unlocking high-fidelity speech recognition and synthesis lies in dramatically increasing the resolution and vocabulary size of these acoustic units.

The Bottleneck of Coarse Quantization

To process audio with the powerful Transformer architectures used in natural language processing, we must first quantize the audio. Traditionally, this involves taking a self-supervised model like HuBERT and using K-means clustering to group similar acoustic frames into a few hundred discrete codes.

However, as Florian Kreyssig and colleagues point out in Biased Self-supervised learning for ASR, these limited cluster counts often fail to capture the full range of phonetic nuances. When the "vocabulary" of sounds is too small, the model is forced to conflate distinct acoustic events. This creates an information bottleneck: the model loses the subtle details necessary to distinguish between similar phonemes or to capture the expressive prosody of a speaker.

Capturing Coarticulation through High-Resolution Targets

Why does a larger vocabulary help? The answer lies in the complexity of human speech, particularly coarticulation—the way the sound of a phoneme changes based on the sounds that come before and after it.

In Exploring Prediction Targets in Masked Pre-Training for Speech Foundation Models, Liwei Chen and Alexander Rudnicky demonstrate that increasing the number of discrete clusters to tens of thousands significantly enhances phoneme recognition. This higher target resolution allows the model to learn specific tokens for these transitional, coarticulated sounds. Instead of a single generic "t" token, a high-granularity model can represent the specific acoustic signature of a "t" followed by an "oo" versus a "t" followed by an "ee."

This principle extends to text-based representations as well. Vasileios Papadourakis and researchers at Amazon Alexa developed Phonetically Induced Subwords (PhIS), showing that when we tokenize text based on how it sounds rather than how it is spelled, the effectiveness of the model scales directly with the size of the phonetic vocabulary.

Disentangling Content from Speaker Identity

One might assume that more clusters would lead to "overfitting" on speaker-specific traits, but the evidence suggests the opposite when handled correctly. Alexander H. Liu and James R. Glass introduced Spin (Speaker-invariant clustering) in Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering. They found that increasing the codebook size actually allows the model to better isolate linguistic content. By providing more "slots" in the vocabulary, the model can use some to capture fine-grained phonetic detail while others are used to normalize across different speakers, leading to more robust representations.

Beyond Single Codebooks: Residual Quantization

For speech synthesis (TTS), the demand for granularity is even higher. A single codebook, no matter how large, often struggles to capture both the high-level semantic meaning and the low-level acoustic texture (like breathiness or high-frequency details).

To solve this, Yichen Han and the team behind QTTS utilize Residual Vector Quantization (RVQ). Instead of one large codebook, they use a hierarchy of codebooks where each layer quantizes the error (the "residual") left by the previous layer. This "Quantize More, Lose Less" philosophy allows the model to layer information:

  1. The first layer captures the core phonetic content.
  2. Subsequent layers add fine-grained acoustic nuances and high-frequency details.

Similarly, Kun Zhou shows in Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis that increasing the resolution of these intermediate tokens directly correlates with improved naturalness and speaker similarity in zero-shot synthesis.

The Sound Alphabet Intuition

The shift toward high-granularity units mirrors the evolution of computer graphics. Just as increasing pixel density (resolution) allows a screen to represent smoother curves and more realistic textures, increasing the density of Acoustic Unit Descriptors (AUDs)—a concept pioneered by Bhiksha Raj in Unsupervised Learning of Acoustic Unit Descriptors—allows models to represent the "high-variance" data of the real world. By expanding the "sound alphabet," we move away from a world of acoustic caricatures toward a precise, digital reconstruction of human speech.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 13 papers
  2. Version 1 · 2026-10-04 · 15 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_task_dependent_unit_granularity_a13cc42b,
  title = {task-dependent unit granularity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-dependent-unit-granularity/rdr_a13cc42b90f946f9853ca68d52adbae5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_the_granularity_of_discrete_acou_a13cc42b_v1,
  title = {Scaling the Granularity of Discrete Acoustic Units},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-the-granularity-of-discrete-acoustic-units/rdr_a13cc42b90f946f9853ca68d52adbae5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}