Bridging the Semantic-Acoustic Gap in Audio Tokenization

This direction is split into other directions: semantic-acoustic representation trade-off. Its last version stays readable and citable here.

Version 1 of 1175 papers260 conceptsfrozen 2026-10-04

The rapid ascent of Large Language Models (LLMs) has sparked a transition from text-only processing to unified Speech Language Models (SpeechLMs). As surveyed by Wenqian Cui and colleagues in Recent Advances in Speech Language Models: A Survey, these models aim to process audio end-to-end, moving beyond the fragmented "ASR + LLM + TTS" pipelines of the past.

However, a fundamental hurdle remains: audio tokenization. To "read" sound, an LLM must convert continuous waves into discrete tokens. Researchers have discovered a persistent conflict between two goals: acoustic fidelity (how natural the sound is) and semantic richness (how much meaning the tokens carry).

The Optimization Dilemma

Traditional neural audio codecs, such as EnCodec or DAC, were designed for compression. They prioritize signal reconstruction—ensuring that the output waveform matches the input as closely as possible. While these codecs produce high-fidelity sound, Zhen Ye and Xu Tan argue in Codec Does Matter that they often lack the linguistic depth needed for understanding. Because they focus on the "texture" of sound, their tokens are often "noisy" and unpredictable for an LLM, leading to hallucinations or word skipping.

Conversely, semantic models like HuBERT or WavLM extract abstract linguistic features. These are excellent for understanding "what" is said but discard "how" it is said—stripping away speaker identity, emotion, and prosody. As Tu Anh Nguyen and Wei-Ning Hsu demonstrate in the Expresso benchmark, using these semantic units for synthesis results in robotic, monotone speech.

Hierarchical Disentanglement

To resolve this, a new generation of "unified" tokenizers has emerged. The core intuition is to disentangle the audio signal into layers.

Xin Zhang and Dong Zhang pioneered this with SpeechTokenizer. Instead of treating all tokens equally, they use a hierarchical approach where the first codebook is forced to align with a "semantic teacher" (like HuBERT), while subsequent codebooks capture the acoustic residuals (the remaining details like pitch and timbre).

This theme is expanded in HASRD by Amir Hussein and Sameer Khurana, which explicitly splits speech into "what" (semantic) and "how" (acoustic) streams. By assigning semantic data to the primary codebook, they achieve a 44% improvement in speech recognition while maintaining high-quality reconstruction.

Semantic-Acoustic Distillation

Another strategy involves distillation, where an acoustic encoder is trained to mimic the representations of a semantic model. In XY-Tokenizer, Luozhijie Jin and colleagues use a dual-tower architecture to mitigate the conflict. They argue that traditional distillation often forces a compromise; by using a two-stage training strategy, they can preserve linguistic accuracy without sacrificing the fine-grained details necessary for emotion and speaker identity.

Similarly, Jeeyoung Yun and Sungwoong Kim introduced LM-SPT, which uses a frozen Whisper encoder to supervise the reconstruction process. This ensures that even at ultra-low bitrates, the tokens remain "LM-aligned"—meaning they behave more like natural language and are easier for generative models to predict.

Measuring the Gap

The community has recently moved toward rigorous benchmarking to quantify this trade-off. AudioCodecBench, led by Lu Wang, and CodecBench, led by Robert H. Deng, both reveal a sobering reality: high-fidelity reconstruction does not guarantee semantic preservation.

A codec might achieve a perfect score in sound quality (like Mel-distance) while completely failing at a downstream task like emotion classification or speech recognition. This suggests that the "best" tokenizer for an LLM isn't the one that sounds the best to a human, but the one that balances the abstract structure of language with the physical reality of sound.

Towards Unified Understanding and Generation

The ultimate goal of this research is a single, unified representation. Models like Ming-UniAudio by Canxiang Yan and UniCodec by Yidi Jiang attempt to bridge this gap by using domain-adaptive codebooks or continuous tokenizers. By mastering both understanding and generation within one framework, these models move us closer to "superhuman" speech AI—systems that understand not just the words we say, but the full acoustic context of our environment.

Papers in this version

Showing 60 of 175 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bridging_the_semantic_acoustic_gap_in_au_25159234,
  title = {Bridging the Semantic-Acoustic Gap in Audio Tokenization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-semantic-acoustic-gap-in-audio-tokenization/rdr_2515923478764c749df02e0c31638dd7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_semantic_acoustic_gap_in_au_25159234_v1,
  title = {Bridging the Semantic-Acoustic Gap in Audio Tokenization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-semantic-acoustic-gap-in-audio-tokenization/rdr_2515923478764c749df02e0c31638dd7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}