TL;DR
LSCodec is a novel discrete speech codec that achieves ultra-low bitrates (down to 0.25 kbps) while explicitly stripping out speaker identity (timbre) from its discrete tokens. By utilizing a single codebook and a multi-stage unsupervised training pipeline with speaker perturbation, it delivers high-fidelity, highly intelligible speech reconstruction and excellent voice conversion capabilities, making it an ideal representation for speech language models.
What Problem Does This Paper Address?
Generative speech language models (LMs) typically rely on converting continuous audio into discrete tokens. Currently, researchers choose between two sub-optimal options:
- Acoustic tokens (e.g., EnCodec, DAC): These capture high-quality audio but require high bitrates and multiple vector quantization (VQ) layers. This introduces complex modeling overhead for speech LMs, such as coarse-to-fine architectures or delay patterns. Furthermore, they repeatedly encode global speaker timbre across every timestep, wasting coding efficiency.
- Semantic tokens (e.g., HuBERT, WavLM): These offer low bitrates by focusing purely on content, but they discard crucial paralinguistic information like prosody (intonation and rhythm), making the generated speech sound flat or unnatural.
LSCodec addresses this dilemma by creating a single-codebook, low-bitrate speech token that explicitly decouples speaker timbre while preserving both linguistic content and time-variant prosody.
Methods
The authors introduce a three-stage unsupervised learning framework that combines speaker perturbation with an information bottleneck to isolate speaker identity from content and prosody.
- Speaker Perturbation: To force the network to discard timbre, the input audio segment is altered using a time-stretching technique (via the WSOLA algorithm). This changes the global pitch and formant positions (timbre) while keeping the duration, content, and prosodic variations intact.
- Stage 1: Speech VAE (Continuous Space): The perturbed audio is passed through a CNN encoder to produce a continuous latent space. To reconstruct the original, unperturbed audio, a Conformer decoder is given the target speaker's true timbre via a separate reference prompt processed by a WavLM model and a position-agnostic cross-attention mechanism. The VAE undergoes multi-task learning: it simultaneously predicts the original mel-spectrogram (to preserve prosody) and SSL semantic tokens (to preserve content).
- Stage 2: Speech VQ-VAE (Discrete Space): A single vector quantization (VQ) layer is injected into the pre-trained VAE. The codebook is initialized using k-means clustering on the continuous latent space, further tightening the information bottleneck to lock out speaker variations.
- Stage 3: Vocoder Training: Finally, a specialized neural vocoder ($\text{CTX-vec2wav}^\alpha$) is trained from scratch to synthesize high-fidelity waveforms directly from the discrete LSCodec tokens and a target speaker prompt.
Datasets
- Training: The model was trained on the LibriTTS corpus, utilizing approximately 360 hours of 24kHz English speech data after filtering out utterances shorter than 6 seconds.
- Evaluation: Evaluation was performed on testset-B (derived from the LibriTTS test-clean split), consisting of 500 utterances from entirely unseen speakers.
Key Insights and Ideas
- Mutual Benefit of Compression and Decoupling: Lowering the token bitrate and removing speaker information are complementary goals. Tightening the information bottleneck naturally sheds global, time-invariant features like timbre.
- Explicit vs. Implicit Bottlenecks: While standard VQ layers provide an implicit bottleneck, they often fail to fully decouple speaker identity on their own. Explicitly damaging the timbre via input perturbation forces the encoder to ignore it entirely.
- The Multi-Task Balance: By forcing the decoder to predict both acoustic features (mel-spectrograms) and semantic features (SSL tokens), the continuous bottleneck is guided to retain the perfect middle ground: rich textual content and expressive prosody, without the speaker's voice profile.
Key Results
The authors evaluated two versions: LSCodec-50Hz (vocabulary size $V=300$, bitrate $0.45\text{ kbps}$) and LSCodec-25Hz (vocabulary size $V=1024$, bitrate $0.25\text{ kbps}$).
Speech Reconstruction
As shown in the reconstruction results table, LSCodec-50Hz outperforms single-codebook and multi-codebook acoustic baselines (like TiCodec and WavTokenizer) in Word Error Rate (WER) and Speaker Similarity (SECS), despite operating at a lower bitrate.
Speaker Decoupling
In speaker probing experiments using an X-vector classifier, tokens from LSCodec showed exceptionally low speaker classification accuracy. This quantitatively proves that speaker timbre is successfully stripped from the discrete representations.
Additionally, in Any-to-Any Voice Conversion (VC) tasks, LSCodec demonstrated superior target speaker similarity (SECS) and pitch contour correlation (P.Corr) compared to other acoustic codecs, validating its clean separation of prosody and timbre.
Limitations and Future Work
While LSCodec achieves state-of-the-art performance for low-bitrate speaker-decoupled coding, the authors note a few areas for improvement:
- Perturbation Techniques: Exploring more sophisticated or aggressive speaker perturbation methods could further improve speaker disentanglement.
- Content Preservation: At ultra-low frame rates (25Hz), there is a slight degradation in intelligibility (higher WER), which could be mitigated with better content-preservation losses.
- Scaling: Testing the framework's scaling capability on much larger, multi-lingual, and noisy datasets remains an open direction for future work.
Summary updated