Large Language Models (LLMs) excel at reasoning because text is a highly compressed, semantically dense representation of information. Speech, by contrast, is a high-bandwidth signal. Traditional neural audio codecs often represent speech at 50Hz or higher, creating a "token expansion" problem: a single sentence that takes 10 text tokens might require 500 audio tokens. This density forces Transformer models to waste their quadratic attention budget on acoustic textures rather than linguistic meaning.
A new wave of research is solving this by shifting toward low-frequency tokenization. By reducing the frame rate of speech tokens to align with the density of text, researchers are enabling LLMs to process audio with the same structural efficiency they apply to written language.
The Problem of Token Expansion
Standard self-supervised speech models like HuBERT or codecs like EnCodec represent audio as dense sequences of frames. As Cheol Jun Cho and colleagues note in Sylber, these high token densities lead to prohibitive computational costs. When a model must attend to 50 tokens for every second of audio, its effective context window shrinks, and it struggles to capture long-range semantic dependencies.
The core insight across recent work is that the "logical" structure of speech is much sparser than its acoustic signal. By "zooming out" the temporal resolution, models can focus on what is being said rather than how the air is vibrating.
Aligning Speech with Text Density
To bridge the gap between modalities, researchers are training models on interleaved speech-text data where the audio tokens mimic the frequency of words. Aohan Zeng, Zhengxiao Du, and their team demonstrated in Scaling Speech-Text Pre-training with Synthetic Interleaved Data that reducing the frame rate to 12.5Hz optimizes the balance between reconstruction and sequence efficiency.
This alignment allows the model to apply pre-trained linguistic reasoning directly to audio. In GLM-4-Voice, this strategy enabled a 9B-parameter LLM to achieve state-of-the-art spoken QA performance, essentially treating speech as just another "dialect" of text.
From Phonemes to Syllables
If 50Hz represents phonetic bits, what does 5Hz represent? Researchers are increasingly looking toward "syllable-like" units as the natural atoms of speech modeling.
- SyllableLM: Alan Baade and David Harwath introduced SyllableLM, which tokenizes speech at just 5–8Hz. This 30x reduction in compute actually improved semantic understanding, suggesting that high-frequency noise often distracts models from high-level linguistic tasks.
- Sylber: Similarly, Sylber distills speech into ~4.3 tokens per second. This approach mirrors "categorical perception" in humans—our brains don't process every micro-fluctuation in sound; we snap them into discrete phonological categories.
The Dual-Representation Strategy
A major challenge of low-frequency tokenization is maintaining audio quality. If you only have 10 tokens per second, how do you capture the speaker's emotion, pitch, and environment? The field is converging on a "dual-representation" paradigm:
- Sparse Tokens for Reasoning: A low-frequency (e.g., 10-12.5Hz) sequence used by the LLM to decide what to say.
- Dense Features for Synthesis: A secondary mechanism to "fill in" the acoustic details.
In BridgeCode, Xiaofen Xing and Xiangmin Xu use 10Hz tokens for efficient generation, backed by dense continuous features for high-fidelity reconstruction. Similarly, Weidong Chen introduced Gogo, which uses a "coarse-to-fine" hierarchy. By focusing the primary autoregressive stage on sparse, semantically dense representations, these models avoid the error accumulation that plagues high-frequency generation.
Pushing the Limits of Semantic Compression
The frontier of this research involves pushing token rates even lower while retaining intelligibility. Georgios Ioannides, Ravid Shwartz-Ziv, and Yann LeCun recently applied the Joint-Embedding Predictive Architecture (JEPA) to speech. Their model operates at an ultra-low 2.5Hz.
At this resolution, tokens are no longer representing sounds; they are representing deep semantic features. This extreme compression allows for massive context windows, enabling models to "read" hours of audio as easily as they read a short essay. As shown by ALMTokenizer and HH-Codec, the goal is no longer just compression for storage, but compression for cognition—creating a representation of speech that is as computationally "cheap" and semantically "rich" as text itself.