Abstract
Neural speech codecs typically rely on Residual Vector Quantization (RVQ) to achieve high compression ratios. However, RVQ often suffers from training instability and information redundancy, particularly when trained on noisy, real-world data. We introduce PURE Codec (Progressive Unfolding of Residual Entropy), a novel framework that utilizes a pre-trained speech enhancement model to anchor the quantization process. By forcing the initial quantization stage to reconstruct a denoised version of the input, the model establishes a stable, low-entropy target that improves overall reconstruction quality and robustness. Our results across diverse datasets show that PURE Codec significantly outperforms standard baselines like the Descript Audio Codec (DAC), maintaining high performance even in extreme noise conditions where traditional models collapse.
Introduction
Modern neural speech codecs compress audio into discrete tokens using multi-stage Residual Vector Quantization (RVQ). In this scheme, the first stage captures a coarse approximation of the signal, and subsequent stages iteratively encode the residual error. While effective, RVQ faces two primary challenges: training instability and redundancy. In noisy environments, the residual signal often becomes chaotic, preventing later stages from learning meaningful patterns and potentially leading to model collapse. Furthermore, without explicit guidance, different stages may encode overlapping information, reducing bitrate efficiency.
PURE Codec addresses these issues by aligning the quantization stages with the natural entropy structure of speech. Since clean speech has lower perceptual entropy than noisy speech, anchoring the first stage to a "clean" core allows the codec to more efficiently distribute information across its codebooks.
Methodology
The core innovation of PURE Codec is enhancement-aware supervision within the RVQ pipeline.
Enhancement-Guided Quantization
During training, the first quantization stream is tasked with reconstructing an enhanced (denoised) version of the input speech, provided by a pre-trained enhancement model. This "low-entropy anchor" ensures the most critical speech information is captured early. Subsequent streams (2 through $L$) then progressively "unfold" the remaining residual entropy, capturing fine details and background noise.
Training Strategy
The model is trained in two stages:
- VAE Pre-training: The encoder and decoder are first trained as a Variational Auto-Encoder to establish a smooth latent space.
- GAN-based Refinement: The full system, including quantizers, is refined using reconstruction losses and adversarial losses from multi-scale discriminators.
A key finding from ablation studies is that the encoder must remain frozen after the VAE stage; fine-tuning it during quantization can lead to significant performance degradation.
Stochastic Scheduling
To maintain flexibility, the model employs a scheduler ($p_{enh}$) that randomly selects between the enhanced and original signal as the target for the first stage during training. This prevents the model from becoming overly dependent on the specific artifacts of the enhancement model.
Experimental Setup
We evaluated PURE Codec across three scenarios representing different data scales and noise levels:
- OWSM-v3.2: A large-scale, multi-domain corpus (150k hours).
- CommonVoice (V13): A crowdsourced dataset with high speaker and quality variability (16k hours).
- URGENT 2024: A dataset specifically designed for robustness, featuring extreme noise and reverberation (2k hours).
All models were benchmarked on the LibriSpeech-test-clean set to assess generalization capabilities.
Results and Discussion
PURE Codec demonstrates superior performance and stability compared to the DAC baseline, particularly in challenging acoustic environments.
Robustness and Stability
On the URGENT dataset, the DAC baseline experienced a performance collapse, yielding a Signal-to-Distortion Ratio (SDR) of -6.79. In contrast, PURE Codec remained stable with an SDR of 1.35 and significantly higher PESQ scores. This stability is attributed to the enhancement guidance providing a consistent "north star" for the first codebook.
Perceptual Entropy and Intelligibility
The authors quantified the benefit of enhancement using Perceptual Entropy (PE), finding that denoising reduces PE by an average of 57.80%. This reduction makes the speech signal significantly more compressible. Consequently, PURE Codec achieves better intelligibility; on the OWSM dataset, it reached a Word Error Rate (WER) of 2.05%, compared to 2.26% for the baseline.
Downstream Applications
When used as a vocabulary for Text-to-Speech (TTS) language models, PURE Codec tokens produced more natural speech (higher UTMOS) and better speaker similarity than baseline tokens, indicating that the learned discrete representations are more semantically and acoustically meaningful.
Conclusion
PURE Codec introduces a principled way to stabilize neural speech codec training by leveraging the entropy structure of audio. While currently optimized for speech through specific enhancement models, the "entropy-guided" paradigm offers a promising path for robust audio compression. Future work will explore extending this framework to general audio domains, including music and environmental sounds.
Summary updated