TL;DR
This paper introduces a novel two-stage training framework for neural audio codecs that jointly handles speech compression and enhancement in noisy environments. By providing a theoretical foundation for this problem, the proposed method achieves superior objective and subjective quality compared to existing state-of-the-art codecs, especially under challenging low-bitrate and high-noise conditions.
What Problem Does This Paper Address?
The core challenge this paper tackles is creating audio codecs that can effectively compress speech while simultaneously cleaning up background noise, all within a single, efficient neural network. Traditional audio codecs often struggle with noisy input, leading to degraded quality. While recent end-to-end neural codecs like SoundStream have shown promise by jointly training compression and enhancement, their training strategies have largely been heuristic. This paper aims to bridge this gap by providing a theoretically grounded approach to optimize this joint task, ensuring both low distortion and high perceptual quality even when the input speech is heavily corrupted by noise.
Methods
The paper's key innovation lies in its two-stage training framework, derived from theoretical analysis of the rate-distortion-perception tradeoff in noisy conditions.
- Theoretical Foundation: The authors first establish a theoretical result showing that an optimal procedure for joint compression and enhancement under noise involves a two-stage optimization. Stage 1 focuses on optimizing an encoder-decoder pair using only distortion loss. Stage 2 then freezes the encoder from Stage 1 and trains a perceptual decoder using a combination of distortion and perception losses. This approach is shown to be optimal for minimizing distortion while achieving high perceptual quality.
- Two-Stage Training Framework (SEStream):
- Stage 1: The encoder-decoder pair is trained using a multi-scale spectral reconstruction loss (a distortion loss) without any discriminators. This stage focuses on achieving good compression and initial noise suppression.
- Stage 2: The encoder and quantizer from Stage 1 are frozen. A new perceptual decoder is trained using a combination of:
- Distortion Loss: The same multi-scale spectral reconstruction loss.
- Adversarial Loss: Using waveform-based and STFT-based discriminators to improve perceptual quality.
- Feature Loss: Comparing internal layer outputs of the discriminators to further refine perceptual quality.
- Codec Architecture: The model utilizes a generator composed of an encoder, a residual vector quantizer (with cascaded vector quantizers for bit-rate scalability), and a decoder. Quantizer dropout is employed for bit-rate flexibility.
Datasets
- Training:
- Clean Speech: LibriTTS [15] (train-clean-100 and train-clean-360), sampled at 24 kHz.
- Noisy Speech: Generated by mixing clean LibriTTS speech with environmental noise from Freesound [16] (music, mechanical noise, natural atmosphere) at varying Signal-to-Noise Ratios (SNRs) from 0 dB to 15 dB. Noise segments were inserted every three seconds.
- Evaluation:
- Clean Test Set: 200 clean samples from the LibriTTS test set.
- Noisy Test Set: The 200 clean samples were corrupted with noise at SNRs of 0 dB, 5 dB, 10 dB, and 15 dB.
- Comparison with other codecs: 20 examples each for low and medium bit-rates, including clean and noisy test sets with SNRs from 0 to 15 dB.
Key Insights and Ideas
- Theoretical Justification for Two-Stage Training: The paper provides a crucial theoretical underpinning for why a two-stage approach is optimal for joint speech compression and enhancement in noisy conditions. This moves beyond heuristic methods and offers a principled way to train such models.
- Trade-off Between Distortion and Perception: The work highlights the inherent trade-off between minimizing distortion and maximizing perceptual quality, especially in the presence of noise. The proposed framework effectively balances these competing objectives.
- Synergy of Compression and Enhancement: By jointly training these tasks, the model can achieve better noise suppression than a standalone enhancement module and better compression quality than a codec trained only on clean speech. The encoder learns to extract features that are robust to noise for compression.
- Importance of Perceptual Loss in Stage 2: While Stage 1 focuses on distortion, Stage 2's use of adversarial and feature losses is critical for generating high-quality, natural-sounding speech, even from noisy inputs.
Key Results
- Objective Quality: SEStream consistently outperforms SoundStream in terms of ViSQOL scores across various bit-rates and noise conditions, showing an average improvement of about 0.057. It also demonstrates better performance on STOI, CSIG, CBAK, and COVL metrics, indicating higher intelligibility and less background noise, particularly at low SNRs (e.g., 0 dB).
- Subjective Quality: MUSHRA scores show that SEStream achieves comparable or slightly better subjective quality than SoundStream, especially at low bit-rates and in noisy conditions. It significantly outperforms traditional codecs like Lyra, EVS, and OPUS in noisy scenarios.
- Bit-rate Scalability: The model supports scalable bit-rates from 3 kbps to 18 kbps by controlling the number of cascaded vector quantizers, similar to SoundStream.
- Qualitative Comparison: Spectrogram analysis shows that SEStream effectively suppresses background noise while retaining more clean speech spectral information compared to SoundStream.
Limitations and Future Work
- Limitations:
- The theoretical analysis primarily focuses on squared error distortion, though the framework is extended to multi-scale spectral reconstruction loss.
- While the denoising flag of SoundStream was enabled for comparison, the paper focuses on its proposed method's inherent joint denoising capability.
- Future Work: Not explicitly detailed in the provided text, but the success in joint compression and enhancement suggests potential for further exploration in real-time applications, diverse noise types, and even more challenging acoustic environments. The theoretical framework could also be extended to other distortion measures or perceptual metrics.
Summary updated