LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec

Yiwei Guo, Zhihan Li, Chenpeng Du, Hankun Wang, Xie Chen, Kai Yu

Abstract

Although discrete speech tokens have exhibited strong potential for language model-based speech generation, their high bitrates and redundant timbre information restrict the development of such models. In this work, we propose LSCodec, a discrete speech codec that has both low bitrate and speaker decoupling ability. LSCodec adopts a three-stage unsupervised training framework with a speaker perturbation technique. A continuous information bottleneck is first established, followed by vector quantization that produces a discrete speaker-decoupled space. A discrete token vocoder finally refines acoustic details from LSCodec. By reconstruction experiments, LSCodec demonstrates superior intelligibility and audio quality with only a single codebook and smaller vocabulary size than baselines. The 25Hz version of LSCodec also achieves the lowest bitrate (0.25kbps) of codecs so far with decent quality. Voice conversion evaluations prove the satisfactory speaker disentanglement of LSCodec, and ablation study further verifies the effectiveness of the proposed training framework.

Introduction

In the field of AI-generated speech, discrete tokens serve as a digital shorthand for audio, enabling the training of Large Language Models (LLMs) for speech tasks. However, current codecs often face a trade-off between fidelity and efficiency. Acoustic codecs provide high fidelity but require high bitrates and multiple token layers, while semantic tokens offer low bitrates but often discard paralinguistic information like emotion and prosody. Furthermore, most codecs redundantly encode speaker timbre into every frame, which is inefficient for modeling.

LSCodec addresses these issues by providing a low-bitrate, single-codebook system that decouples speaker identity from speech content. By isolating timbre as a global characteristic, LSCodec retains the essential content and prosody of speech at bitrates as low as 0.25 kbps.

Methodology

LSCodec is developed through a three-stage unsupervised pipeline designed to explicitly separate speaker-dependent and speaker-independent features.

The three-stage unsupervised training framework of LSCodec, illustrating the progression from continuous VAE to discrete VQ-VAE and final vocoder refinement.

Stage 1: Speech VAE with Speaker Perturbation

The model is initially trained as a Variational Autoencoder (VAE). To force the bottleneck to ignore speaker identity, a speaker perturbation technique is applied to the input audio using the WSOLA algorithm for time-stretching and pitch-shifting. While the encoder sees this perturbed audio, the decoder is provided with a separate "timbre reference" via cross-attention. This architecture ensures the latent space primarily stores content and prosody.

Stage 2: Speech VQ-VAE

The continuous latent space from the first stage is quantized into a discrete codebook. The system uses k-means clustering for codebook initialization, ensuring that the discrete tokens effectively represent the most critical acoustic features.

Stage 3: Vocoder Refinement

A specialized vocoder, based on the CTX-vec2wav architecture, is trained to translate the discrete LSCodec tokens back into high-fidelity 24kHz audio. This stage ensures that the final output maintains high acoustic quality despite the aggressive compression.

Key Results and Insights

LSCodec was trained on the LibriTTS corpus and evaluated using the test-clean set to ensure generalization to unseen speakers.

  • Intelligibility and Quality: The 50Hz version of LSCodec (0.45 kbps) achieved a Word Error Rate (WER) of 3.33%, significantly outperforming other low-bitrate codecs such as TiCodec (10.44%). The 25Hz version achieved a record-low bitrate of 0.25 kbps while maintaining decent audio quality.
  • Speaker Decoupling: Probing experiments demonstrated that LSCodec successfully removes speaker information from its tokens. This decoupling enables high-quality "Any-to-Any" voice conversion, achieving a Speaker Similarity Score (SECS) of 0.853 when combining source tokens with a target timbre prompt.
  • Prosody Preservation: Unlike pure semantic models, LSCodec retains the "musicality" of speech by incorporating a mel-spectrogram reconstruction task alongside semantic token prediction during training.

Conclusion

LSCodec represents a significant step toward efficient speech representation for language modeling. By utilizing a single codebook and explicit speaker perturbation, it achieves superior intelligibility at ultra-low bitrates. While the current non-causal architecture is not yet optimized for real-time streaming, the framework provides a robust foundation for future research into speaker-independent speech synthesis and compression.

Summary updated

Abstract

Although discrete speech tokens have exhibited strong potential for language model-based speech generation, their high bitrates and redundant timbre information restrict the development of such models. In this work, we propose LSCodec, a discrete speech codec that has both low bitrate and speaker decoupling ability. LSCodec adopts a three-stage unsupervised training framework with a speaker perturbation technique. A continuous information bottleneck is first established, followed by vector quantization that produces a discrete speaker-decoupled space. A discrete token vocoder finally refines acoustic details from LSCodec. By reconstruction experiments, LSCodec demonstrates superior intelligibility and audio quality with only a single codebook and smaller vocabulary size than baselines. The 25Hz version of LSCodec also achieves the lowest bitrate (0.25kbps) of codecs so far with decent quality. Voice conversion evaluations prove the satisfactory speaker disentanglement of LSCodec, and ablation study further verifies the effectiveness of the proposed training framework.
BibTeX
@inproceedings{guo2025lscodec,
  author = {Yiwei Guo and Zhihan Li and Chenpeng Du and Hankun Wang and Xie Chen and Kai Yu},
  title = {{LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec}},
  booktitle = {Interspeech},
  year = {2025},
  doi = {10.21437/interspeech.2025-1106},
}
Figure