In text processing, tokenization is relatively straightforward: words or subwords map cleanly to discrete vocabulary IDs. In audio, however, the signal is continuous, high-dimensional, and packed with overlapping information—combining phonetic content, speaker identity, emotional prosody, and background noise.
To process audio using modern transformer architectures, researchers discretize continuous audio features into a finite set of "audio tokens" or "acoustic units" via vector quantization. But how large should this discrete vocabulary be, and how long should each token last?
A growing body of research demonstrates that the granularity of these discrete units—defined by both the size of the quantization codebook and the temporal duration of the units—is a critical lever for model performance. Optimizing this granularity is not just about increasing resolution; it is about matching the token's information density to the specific demands of the downstream task.
The Cost of Low Resolution: Why Small Codebooks Conflate Signals
When quantizing continuous speech features into discrete tokens, a model maps continuous vectors to the nearest cluster centroid in a codebook. If the codebook is too small, distinct acoustic phenomena are forced to share the same token. This creates an information bottleneck where subtle but vital variations are permanently lost.
Continuous Feature Space Small Codebook (Low Granularity)
[ Vector A ] ──┐
├───────► [ Centroid 1 ] (Conflated Token)
[ Vector B ] ──┘
Continuous Feature Space Large Codebook (High Granularity)
[ Vector A ] ──────────► [ Centroid 1 ] (Distinct Token)
[ Vector B ] ──────────► [ Centroid 2 ] (Distinct Token)
To prevent this conflation, researchers have scaled up codebook sizes. Liu et al. (2023) showed that increasing the size of the discrete codebook allows self-supervised speech models to capture much finer phonetic nuances, leading to significantly better acoustic unit discovery. When codebooks scale up to massive sizes—such as the 65,536-entry codebook enabled by the dual-decoding architecture of Guo et al. (2024)—the expanded vocabulary directly translates into higher naturalness and intelligibility in large language model frameworks, yielding substantial reductions in character error rates.
This high-resolution quantization is particularly vital for capturing paralinguistic details. As Yang et al. (2026) discovered, fine-grained discrete units retain rich paralinguistic information like emotion and speaker identity. These nuances are completely smoothed away by coarser clustering methods, proving that high-cardinality quantization is essential for preserving the multi-dimensional nature of speech.
Disentangling Content from the Environment
A larger, more granular token space does more than just preserve detail; it acts as a high-dimensional filter that allows orthogonal properties of the audio signal to separate.
Consider the challenge of background-preserving voice conversion, where a model must alter a speaker's voice while leaving background music or noise untouched. Liu et al. (2025) demonstrated that expanding a semantic token codebook from 500 to 2,000 clusters enhances the model's ability to disentangle phonemes from background noise. Because the higher-resolution space provides dedicated clusters for pure speech versus noise-contaminated speech, the model avoids conflating linguistic content with environmental audio elements.
This capacity to represent high-variance data extends across several domains:
- Environmental Audio: Raj et al. (2011) found that expanding the vocabulary of latent Acoustic Unit Descriptors significantly enhances a system's ability to represent complex, unstructured background noise and real-world video events.
- Cross-Modal Alignment: In speech-text LLMs, Shaik et al. (2024) showed that increasing signal discretization granularity preserves the fine acoustic and phonetic nuances, such as tone, required for effective cross-modal learning.
- Accent Simulation: Onda et al. (2024) discovered that scaling the number of discrete units allows generative models to simulate foreign accents more naturally, as a larger vocabulary smooths out fine-grained phonetic substitutions.
Task-Dependent Granularity: Balancing Time and Vocabulary
Granularity is not a one-size-fits-all metric. It is defined by a crucial interplay between two dimensions: spatial granularity (the size of the codebook) and temporal granularity (the duration of each token).
Visser and Kamper (2025) explicitly mapped this trade-off, demonstrating that the optimal granularity is highly task-dependent. They found that coarser acoustic units—tokens that represent longer segments of time—are inherently detrimental for low-level phonetic discrimination tasks because they blur rapid acoustic transitions. However, for higher-level tasks like speech synthesis and language modeling, these longer-duration units are highly beneficial because they reduce the sequence length the transformer must model. Crucially, the authors noted that these coarser temporal units must be paired with larger codebooks to maintain performance; a large vocabulary compensates for the longer time step by providing a richer descriptive alphabet.
Task Type Optimal Token Configuration
┌──────────────────────────┐
│ Phonetic Discrimination │ ► Fine-grained temporal units (Short duration)
└──────────────────────────┘
┌──────────────────────────┐
│ Speech Synthesis & LM │ ► Coarser temporal units (Long duration) + Large Codebook
└──────────────────────────┘
This task-dependent optimization explains why different architectures achieve success at different vocabulary scales. For instance, in paralinguistic classification, Gosztolya (2019) demonstrated that Bag-of-Audio-Words frameworks steadily improve as the audio-word vocabulary becomes finer and more discriminative. Similarly, Karamanolakis et al. (2016) showed that expanding the discrete auditory alphabet directly enhances a model's capacity to capture subtle semantic differences in audio-based distributional representations.
For automatic speech recognition (ASR), Papadourakis et al. (2021) showed that tokenization based on phonetic subwords yields greater relative improvements over standard text-based methods as the vocabulary size grows, capturing richer acoustic-linguistic information. This aligns with the foundational principle articulated by Siniscalchi et al. (2009), who argued that designing a broad set of detailed articulatory categories to maximize acoustic resolution provides far greater discriminative power for spoken language recognition than relying on a smaller, simpler feature set.
Ultimately, by decoupling these layers of granularity, modern architectures can optimize for both content and style. For example, the text-to-speech framework developed by Zhou et al. (2024) uses a two-stage approach: an autoregressive model first predicts phonetically rich tokens, which a second model then maps to high-resolution acoustic clusters. By scaling the cluster count in the final stage, the model captures the fine prosodic and phonetic nuances necessary to achieve exceptional zero-shot speaker naturalness.
Key Foundational Work
Unsupervised Learning of Acoustic Unit Descriptors for Audio Content Representation and Classification (2011) — contributed 1 concept elements to this direction.
Exploring universal attribute characterization of spoken languages for spoken language recognition (2009) — contributed 1 concept elements to this direction.