The rapid ascent of Large Language Models (LLMs) has sparked a transition from text-only processing to unified Speech Language Models (SpeechLMs). As surveyed by Wenqian Cui and colleagues in Recent Advances in Speech Language Models: A Survey, these models aim to process audio end-to-end, moving beyond the fragmented "ASR + LLM + TTS" pipelines of the past.
However, a fundamental hurdle remains: audio tokenization. To "read" sound, an LLM must convert continuous waves into discrete tokens. Researchers have discovered a persistent conflict between two goals: acoustic fidelity (how natural the sound is) and semantic richness (how much meaning the tokens carry).
The Optimization Dilemma
Traditional neural audio codecs, such as EnCodec or DAC, were designed for compression. They prioritize signal reconstruction—ensuring that the output waveform matches the input as closely as possible. While these codecs produce high-fidelity sound, Zhen Ye and Xu Tan argue in Codec Does Matter that they often lack the linguistic depth needed for understanding. Because they focus on the "texture" of sound, their tokens are often "noisy" and unpredictable for an LLM, leading to hallucinations or word skipping.
Conversely, semantic models like HuBERT or WavLM extract abstract linguistic features. These are excellent for understanding "what" is said but discard "how" it is said—stripping away speaker identity, emotion, and prosody. As Tu Anh Nguyen and Wei-Ning Hsu demonstrate in the Expresso benchmark, using these semantic units for synthesis results in robotic, monotone speech.
Hierarchical Disentanglement
To resolve this, a new generation of "unified" tokenizers has emerged. The core intuition is to disentangle the audio signal into layers.
Xin Zhang and Dong Zhang pioneered this with SpeechTokenizer. Instead of treating all tokens equally, they use a hierarchical approach where the first codebook is forced to align with a "semantic teacher" (like HuBERT), while subsequent codebooks capture the acoustic residuals (the remaining details like pitch and timbre).
This theme is expanded in HASRD by Amir Hussein and Sameer Khurana, which explicitly splits speech into "what" (semantic) and "how" (acoustic) streams. By assigning semantic data to the primary codebook, they achieve a 44% improvement in speech recognition while maintaining high-quality reconstruction.
Semantic-Acoustic Distillation
Another strategy involves distillation, where an acoustic encoder is trained to mimic the representations of a semantic model. In XY-Tokenizer, Luozhijie Jin and colleagues use a dual-tower architecture to mitigate the conflict. They argue that traditional distillation often forces a compromise; by using a two-stage training strategy, they can preserve linguistic accuracy without sacrificing the fine-grained details necessary for emotion and speaker identity.
Similarly, Jeeyoung Yun and Sungwoong Kim introduced LM-SPT, which uses a frozen Whisper encoder to supervise the reconstruction process. This ensures that even at ultra-low bitrates, the tokens remain "LM-aligned"—meaning they behave more like natural language and are easier for generative models to predict.
Measuring the Gap
The community has recently moved toward rigorous benchmarking to quantify this trade-off. AudioCodecBench, led by Lu Wang, and CodecBench, led by Robert H. Deng, both reveal a sobering reality: high-fidelity reconstruction does not guarantee semantic preservation.
A codec might achieve a perfect score in sound quality (like Mel-distance) while completely failing at a downstream task like emotion classification or speech recognition. This suggests that the "best" tokenizer for an LLM isn't the one that sounds the best to a human, but the one that balances the abstract structure of language with the physical reality of sound.
Towards Unified Understanding and Generation
The ultimate goal of this research is a single, unified representation. Models like Ming-UniAudio by Canxiang Yan and UniCodec by Yidi Jiang attempt to bridge this gap by using domain-adaptive codebooks or continuous tokenizers. By mastering both understanding and generation within one framework, these models move us closer to "superhuman" speech AI—systems that understand not just the words we say, but the full acoustic context of our environment.