LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec

Yiwei Guo, Zhihan Li, Chenpeng Du, Hankun Wang, Xie Chen, Kai Yu

TL;DR

LSCodec is a novel discrete speech codec that achieves ultra-low bitrates (down to 0.25 kbps) while explicitly stripping out speaker identity (timbre) from its discrete tokens. By utilizing a single codebook and a multi-stage unsupervised training pipeline with speaker perturbation, it delivers high-fidelity, highly intelligible speech reconstruction and excellent voice conversion capabilities, making it an ideal representation for speech language models.

What Problem Does This Paper Address?

Generative speech language models (LMs) typically rely on converting continuous audio into discrete tokens. Currently, researchers choose between two sub-optimal options:

  1. Acoustic tokens (e.g., EnCodec, DAC): These capture high-quality audio but require high bitrates and multiple vector quantization (VQ) layers. This introduces complex modeling overhead for speech LMs, such as coarse-to-fine architectures or delay patterns. Furthermore, they repeatedly encode global speaker timbre across every timestep, wasting coding efficiency.
  2. Semantic tokens (e.g., HuBERT, WavLM): These offer low bitrates by focusing purely on content, but they discard crucial paralinguistic information like prosody (intonation and rhythm), making the generated speech sound flat or unnatural.

LSCodec addresses this dilemma by creating a single-codebook, low-bitrate speech token that explicitly decouples speaker timbre while preserving both linguistic content and time-variant prosody.

Methods

The authors introduce a three-stage unsupervised learning framework that combines speaker perturbation with an information bottleneck to isolate speaker identity from content and prosody.

Architecture and Training Diagram of LSCodec
  1. Speaker Perturbation: To force the network to discard timbre, the input audio segment is altered using a time-stretching technique (via the WSOLA algorithm). This changes the global pitch and formant positions (timbre) while keeping the duration, content, and prosodic variations intact.
  2. Stage 1: Speech VAE (Continuous Space): The perturbed audio is passed through a CNN encoder to produce a continuous latent space. To reconstruct the original, unperturbed audio, a Conformer decoder is given the target speaker's true timbre via a separate reference prompt processed by a WavLM model and a position-agnostic cross-attention mechanism. The VAE undergoes multi-task learning: it simultaneously predicts the original mel-spectrogram (to preserve prosody) and SSL semantic tokens (to preserve content).
  3. Stage 2: Speech VQ-VAE (Discrete Space): A single vector quantization (VQ) layer is injected into the pre-trained VAE. The codebook is initialized using k-means clustering on the continuous latent space, further tightening the information bottleneck to lock out speaker variations.
  4. Stage 3: Vocoder Training: Finally, a specialized neural vocoder ($\text{CTX-vec2wav}^\alpha$) is trained from scratch to synthesize high-fidelity waveforms directly from the discrete LSCodec tokens and a target speaker prompt.

Datasets

  • Training: The model was trained on the LibriTTS corpus, utilizing approximately 360 hours of 24kHz English speech data after filtering out utterances shorter than 6 seconds.
  • Evaluation: Evaluation was performed on testset-B (derived from the LibriTTS test-clean split), consisting of 500 utterances from entirely unseen speakers.

Key Insights and Ideas

  • Mutual Benefit of Compression and Decoupling: Lowering the token bitrate and removing speaker information are complementary goals. Tightening the information bottleneck naturally sheds global, time-invariant features like timbre.
  • Explicit vs. Implicit Bottlenecks: While standard VQ layers provide an implicit bottleneck, they often fail to fully decouple speaker identity on their own. Explicitly damaging the timbre via input perturbation forces the encoder to ignore it entirely.
  • The Multi-Task Balance: By forcing the decoder to predict both acoustic features (mel-spectrograms) and semantic features (SSL tokens), the continuous bottleneck is guided to retain the perfect middle ground: rich textual content and expressive prosody, without the speaker's voice profile.

Key Results

The authors evaluated two versions: LSCodec-50Hz (vocabulary size $V=300$, bitrate $0.45\text{ kbps}$) and LSCodec-25Hz (vocabulary size $V=1024$, bitrate $0.25\text{ kbps}$).

Speech Reconstruction

As shown in the reconstruction results table, LSCodec-50Hz outperforms single-codebook and multi-codebook acoustic baselines (like TiCodec and WavTokenizer) in Word Error Rate (WER) and Speaker Similarity (SECS), despite operating at a lower bitrate.

Table 1: Reconstruction results. Bitrates are presented in kbps. Parentheses denote model versions, and “-” means not compa- rable because of sampling rate.

Speaker Decoupling

In speaker probing experiments using an X-vector classifier, tokens from LSCodec showed exceptionally low speaker classification accuracy. This quantitatively proves that speaker timbre is successfully stripped from the discrete representations.

Figure 2: Accuracy curve in speaker probing experiments.

Additionally, in Any-to-Any Voice Conversion (VC) tasks, LSCodec demonstrated superior target speaker similarity (SECS) and pitch contour correlation (P.Corr) compared to other acoustic codecs, validating its clean separation of prosody and timbre.

Limitations and Future Work

While LSCodec achieves state-of-the-art performance for low-bitrate speaker-decoupled coding, the authors note a few areas for improvement:

  • Perturbation Techniques: Exploring more sophisticated or aggressive speaker perturbation methods could further improve speaker disentanglement.
  • Content Preservation: At ultra-low frame rates (25Hz), there is a slight degradation in intelligibility (higher WER), which could be mitigated with better content-preservation losses.
  • Scaling: Testing the framework's scaling capability on much larger, multi-lingual, and noisy datasets remains an open direction for future work.

Summary updated

Abstract

Although discrete speech tokens have exhibited strong potential for language model-based speech generation, their high bitrates and redundant timbre information restrict the development of such models. In this work, we propose LSCodec, a discrete speech codec that has both low bitrate and speaker decoupling ability. LSCodec adopts a multi-stage unsupervised training framework with a speaker perturbation technique. A continuous information bottleneck is first established, followed by vector quantization that produces a discrete speaker-decoupled space. A discrete token vocoder finally refines acoustic details from LSCodec. By reconstruction evaluations, LSCodec demonstrates superior intelligibility and audio quality with only a single codebook and smaller vocabulary size than baselines. Voice conversion and speaker probing experiments prove the excellent speaker disentanglement of LSCodec, and ablation study verifies the effectiveness of the proposed training framework.
BibTeX
@inproceedings{guo2025lscodec,
  author = {Yiwei Guo and Zhihan Li and Chenpeng Du and Hankun Wang and Xie Chen and Kai Yu},
  title = {{LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec}},
  booktitle = {Interspeech},
  year = {2025},
  doi = {10.21437/interspeech.2025-1106},
}
Figure