ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling

Dongchao Yang, Songxiang Liu, Haohan Guo, Jiankun Zhao, Yuanyuan Wang, Helin Wang, Zeqian Ju, Xubo Liu, Xueyuan Chen, Xu Tan, Xixin Wu, Helen Meng

TL;DR

The paper introduces ALMTokenizer, a novel audio codec tokenizer designed specifically to bridge the gap between low-bitrate compression and semantic richness in audio language models (LLMs). By utilizing a query-based compression strategy and integrating semantic priors directly into the quantization layer, it achieves state-of-the-art audio reconstruction at ultra-low bitrates (0.41 kbps). When plugged into an audio language model, ALMTokenizer significantly boosts performance across both audio understanding and generation tasks.

What Problem Does This Paper Address?

Audio language models typically process audio by converting continuous waveforms into discrete tokens. Currently, researchers rely on two types of tokenizers, each with a distinct trade-off:

  1. Acoustic tokenizers (e.g., Encodec, DAC): Excel at capturing high-fidelity acoustic details (timbre, environment) but operate frame-by-frame, ignoring broader cross-frame context. This results in high bitrates and tokens that are difficult for language models to predict efficiently.
  2. Semantic tokenizers (e.g., HuBERT, WavLM): Capture linguistic meaning effectively, making them easy for language models to process, but they discard crucial acoustic details, requiring complex, slow secondary decoders (like diffusion models) to reconstruct actual audio.

The authors identify a critical sweet spot for audio language modeling: a tokenizer must be both low-bitrate (for training and inference efficiency) and semantically rich (for lower perplexity and better generation quality).

Figure 1. The performance comparison when different types of tokenizer is used for audio modeling. PPL refers to perplexity.

Methods

ALMTokenizer departs from traditional frame-by-frame downsampling by introducing several novel architectural and loss-based techniques:

  • Query-based Audio Compression: Instead of treating every audio frame equally, the model interleaves learnable "query tokens" within the audio frame sequence. A Transformer encoder models the global context across frames, and the query tokens compress this holistic information before it passes to the Residual Vector Quantization (RVQ) layer. This allows dynamic control over the compression rate by simply adjusting the query window size ($w$).
  • Semantic Priors for VQ Initialization: To force the quantization space to retain meaning, the authors initialize the RVQ codebook using $k$-means cluster centers derived from pre-trained self-supervised models: Wav2vec2 for speech and BEATs for general sounds.
  • Masked Autoencoder (MAE) Loss: During training, a portion of the audio frames is masked (20–30%). The model is forced to reconstruct the missing frames, incentivizing the network to capture global semantic structures rather than just local acoustic patterns.
  • Autoregressive (AR) Prediction Loss: The authors observed that language models struggle to predict the residual tokens in deeper VQ layers. To fix this distribution mismatch, they introduce a lightweight continuous AR transformer that predicts deeper-layer tokens from shallower ones during tokenizer training.

The model is trained using a Two-stage Training Strategy: Stage 1 trains an unquantized AutoEncoder with MAE loss to establish global semantic representations. Stage 2 freezes the initial processing layers and trains the full quantized network using a combination of GAN, MAE, and AR losses.

Figure 2. The left part illustrates the framework of the previous audio codec, while the right part provides an overview of the proposed ALMTokenizer. w denotes the window size. The details of ALMTokenizer can be found in Section 3.2.

Datasets

The tokenizer was trained on approximately 4,500 hours of diverse audio data:

  • Speech: LibriTTS and a 2,000-hour subset of Multilingual LibriSpeech (MLS).
  • Sound: A 1,000-hour subset of AudioSet.
  • Music: A 1,000-hour subset of the Million Song Dataset.

For downstream Audio Language Model evaluation, the authors utilized LibriHeavy (speech), AudioCaps and BBC Sound Effects (sound), and LP-MusicCaps (music). Evaluation benchmarks included VCTK (speech), AudioCaps validation (sound), and MusicCaps (music).

Key Insights and Ideas

  • Context is Key for Compression: Traditional codecs compress each frame independently. By letting a Transformer look across frames via query tokens, ALMTokenizer achieves massive compression gains without sacrificing the underlying message of the audio.
  • Fixing the Residual Mismatch: Language models inherently struggle with the hierarchical nature of RVQ tokens (where layer 1 is coarse and layers 2–3 are fine details). Introducing an explicit AR prediction loss in the latent space aligns the tokenizer's structure with how an autoregressive language model actually learns.
  • Codebook Initialization Matters: Simply letting a codebook learn from scratch often leads to purely acoustic representations. Seeding it with semantic clusters from Wav2vec2 and BEATs permanently bakes semantic awareness into the discrete token space.

Key Results

ALMTokenizer was evaluated against top-tier baselines like Encodec, DAC, and Wavtokenizer.

  • Ultra-low Bitrate Fidelity: At just 0.41 kbps (12.5 Hz frame rate with 3 VQ layers), ALMTokenizer achieved a speech reconstruction UT-MOS score of 3.76 and a ViSQOL score of 3.78, outperforming MimiCodec and Wavtokenizer at similar or higher bitrates.
  • Superior Language Modeling Performance: When integrated into a Moshi-style audio language model framework, ALMTokenizer yielded dramatic improvements in text-to-speech (TTS) robustness. It achieved a Word Error Rate (WER) of 11.7% on LibriSpeech test-clean, vastly outperforming Encodec (22.9%) and DAC (24.5%).
  • General Sound & Music Mastery: Unlike speech-only tokenizers, ALMTokenizer demonstrated strong classification and reconstruction capabilities on environmental sounds and music, outperforming specialized baselines in downstream audio/music captioning tasks.
Figure 3. The performance comparison with or without AR loss.

Limitations and Future Work

Despite its strong performance, the authors note a few limitations:

  • Complex Audio Reconstruction: Reconstructing highly intricate soundscapes and dense music at ultra-low bitrates remains a steep challenge, leaving room for audio fidelity improvements.
  • Semantic Gap: While semantically richer than previous codecs, it still slightly trails pure self-supervised models (like HuBERT) on pure speech recognition tasks.
  • Compute Waste: The two-stage training pipeline discards modules like the MAE encoder/decoder after the first stage. Future work will focus on recycling these components to initialize downstream language modeling tasks more efficiently.

Summary updated

Abstract

Recent advancements in audio language models have underscored the pivotal role of audio tokenization, which converts audio signals into discrete tokens, thereby facilitating the application of language model architectures to the audio domain. In this study, we introduce ALMTokenizer, a novel low-bitrate and semantically rich audio codec tokenizer for audio language models. Prior methods, such as Encodec, typically encode individual audio frames into discrete tokens without considering the use of context information across frames. Unlike these methods, we introduce a novel query-based compression strategy to capture holistic information with a set of learnable query tokens by explicitly modeling the context information across frames. This design not only enables the codec model to capture more semantic information but also encodes the audio signal with fewer token sequences. Additionally, to enhance the semantic information in audio codec models, we introduce the following: (1) A masked autoencoder (MAE) loss, (2) Vector quantization based on semantic priors, and (3) An autoregressive (AR) prediction loss. As a result, ALMTokenizer achieves competitive reconstruction performance relative to state-of-the-art approaches while operating at a lower bitrate. Within the same audio language model framework, ALMTokenizer outperforms previous tokenizers in audio understanding and generation tasks.
BibTeX
@inproceedings{yang2025almtokenizer,
  author = {Dongchao Yang and Songxiang Liu and Haohan Guo and Jiankun Zhao and Yuanyuan Wang and Helin Wang and Zeqian Ju and Xubo Liu and Xueyuan Chen and Xu Tan and Xixin Wu and Helen Meng},
  title = {{ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=QtMcq04xXd},
}
Figure