LMCodec: A Low Bitrate Speech Codec with Causal Transformer Models

Teerapat Jenrungrot, Michael Chinen, W. Bastiaan Kleijn, Jan Skoglund, Zalán Borsos, Neil Zeghidour, Marco Tagliasacchi

Introduction

In the field of telecommunications, speech coding is the essential process of compressing voice signals to fit through limited digital bandwidth. While industry-standard codecs like Opus and EVS perform exceptionally well at medium bitrates, their quality degrades significantly when pushed below 3 kbps, often resulting in robotic, muffled, or distorted audio.

LMCodec addresses this gap by reframing audio compression as a generative modeling problem rather than a traditional signal processing task. By combining causal convolutional encoders with Transformer-based generative modeling, LMCodec achieves high-quality audio at ultra-low bitrates (1–1.5 kbps), effectively providing a 10x improvement in compression efficiency compared to Opus at 12 kbps.

System Architecture

LMCodec builds upon the SoundStream architecture, integrating autoregressive Transformers to enhance its predictive capabilities. The system operates through a multi-stage pipeline designed for high fidelity and low latency.

1. The SoundStream Backbone

The foundation of the system is a causal convolutional encoder that transforms raw audio into a hierarchy of tokens using Residual Vector Quantization (RVQ). These tokens are organized from "coarse" (capturing the primary linguistic structure) to "fine" (capturing the acoustic texture).

2. Conditional Entropy Coding

To maximize compression, LMCodec utilizes a Transformer to model the probability distribution of the next token based on previous ones. This enables lossless entropy coding; because the model can accurately predict many tokens, they require almost zero space in the bitstream, further driving down the bitrate.

3. Generative Synthesis

The "secret sauce" of the system is its approach to transmission. To save bandwidth, the sender transmits only the coarse tokens. On the receiver side, a second "fine-level" Transformer predicts—or "hallucinates"—the missing fine tokens based on the coarse data received. This allows the system to reconstruct high-fidelity audio from a minimal "skeleton" of information.

4. Causality

Crucially, the entire architecture is causal. The model never looks at "future" audio frames to encode the current one, making it suitable for real-time, low-latency applications such as VoIP and mobile telephony.

Experimental Setup and Datasets

The researchers utilized several large-scale standard datasets to train and evaluate the system:

  • LibriVox & Libri-Light: Used for training the base SoundStream model and the AudioLM Transformer components over 1 million steps.
  • LibriSpeech (Clean Test Split): Used for objective evaluations, including Word Error Rate (WER) and Character Error Rate (CER).
  • VCTK: Used for subjective MUSHRA-like listening tests to compare LMCodec against established codecs.

Key Insights

  • Predictability as Compression: By treating speech as a "language" of audio tokens, the system leverages the inherent redundancy of human voice. If a model can guess the next token, that token does not need to be transmitted.
  • Semantic vs. Acoustic Layers: The researchers found that the initial layers of RVQ tokens contain the majority of semantic and linguistic information. By prioritizing these and letting the receiver's AI fill in the acoustic "skin," the system achieves massive savings.
  • The "Babbling" Threshold: At extremely low bitrates (e.g., using only one quantizer), the model may produce speech-like sounds that lack coherent words, a phenomenon known as "babbling." However, increasing the bitrate to ~1 kbps (adding a few more quantizers) stabilizes the linguistic content and ensures intelligibility.

Results and Performance

LMCodec demonstrates a significant leap in efficiency and quality:

  • Subjective Quality: In MUSHRA listening tests, LMCodec at 1.1 kbps was rated significantly higher than Opus at 12 kbps and SoundStream at 3 kbps.
  • Objective Metrics: Word Error Rates (WER) remain remarkably low even at ultra-low bitrates, indicating that the generative process preserves the semantic integrity of the original speech.

Limitations and Future Work

While LMCodec represents a breakthrough in speech coding, several challenges remain:

  • Computational Complexity: Transformers are more computationally intensive than traditional codecs, requiring further optimization for mobile and edge hardware.
  • Non-Speech Audio: The model is specifically tuned for human speech. Its performance on background music or complex environmental noise—which is less predictable than language—is currently limited.
  • Silence Artifacts: The model occasionally loses confidence during transitions or periods of silence, which can lead to minor audible artifacts.

Conclusion

LMCodec demonstrates that generative AI can revolutionize audio compression. By moving beyond traditional signal processing and leveraging the predictive power of Transformers, it is possible to maintain high-quality communication at bitrates previously thought impossible for clear speech.

Summary updated

BibTeX
@inproceedings{jenrungrot2023lmcodec,
  author = {Teerapat Jenrungrot and Michael Chinen and W. Bastiaan Kleijn and Jan Skoglund and Zalán Borsos and Neil Zeghidour and Marco Tagliasacchi},
  title = {{LMCodec: A Low Bitrate Speech Codec with Causal Transformer Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095442},
}
Figure