TL;DR
This paper introduces PERL, a novel framework for speech denoising that uses an ensemble of pre-trained models to enforce perceptual quality. By leveraging perceptual losses derived from various audio tasks, PERL significantly improves denoising performance, outperforming state-of-the-art methods, especially when using acoustic event classification and self-supervised models.
What Problem Does This Paper Address?
Traditional deep learning models for speech denoising often struggle to improve the perceptual quality of the enhanced audio. While objective metrics like PESQ exist, they are expensive to use for evaluation and can be difficult to optimize directly. Existing methods that use perceptual losses (deep feature losses) can improve quality but often focus on specific speech properties (e.g., speaker identity). This paper aims to address the challenge of achieving better overall perceptual and intelligibility metrics by exploring a more comprehensive approach that combines multiple perceptual loss signals from diverse pre-trained models. The authors also investigate how to best utilize these diverse signals, especially when dealing with limited training data and potential domain mismatches between the pre-trained models and the denoising task.
Methods
The core contribution is the Perceptual Ensemble Regularization Loss (PERL) framework. PERL builds upon a strong baseline denoising model (a Conformer Transformer Network) and augments its training objective with multiple perceptual losses. These perceptual losses are derived from the activation spaces of frozen, pre-trained auxiliary networks. The paper explores six such networks:
- Speaker Classification: Enforces preservation of speaker identity.
- Acoustic Model (DeepSpeech2): Focuses on preserving speech content.
- Speaker Embedding (RawNet2): Another approach to speaker identity preservation.
- Emotion Classification (IEMOCAP): Aims to retain emotional nuances.
- Self-Supervised Speech Encoders (PASE+, wav2vec 2.0): These models are trained on pretext tasks and are hypothesized to capture general speech representations that are robust to noise.
PERL combines these perceptual losses with a standard l₁ loss in the STFT domain. The authors investigate using these auxiliary models individually and in combination, exploring different weighting strategies for the combined losses. They also analyze the effectiveness of various Multi-Task Learning (MTL) weight learning methods versus hand-tuned weights.
Datasets
- VCTK-DEMAND: This dataset is used for training and evaluation of the speech denoising system. It's described as a 16KHz small corpus with fixed training (10 hours, 30 speakers) and validation (30 minutes, 2 speakers) splits.
- AudioSet: Used to train the Acoustic Event Classification model.
- LibriSpeech: Used to train the Acoustic Model (DeepSpeech2) and the self-supervised speech encoders (PASE+, wav2vec 2.0).
- VoxCeleb: Used to train the Speaker Embedding model (RawNet2).
- IEMOCAP: Used to train the Speech Emotion Classification model.
Key Insights and Ideas
- Ensemble of Perceptual Losses is Powerful: Combining multiple perceptual loss signals from diverse pre-trained models can significantly boost denoising performance beyond single-loss approaches.
- Self-Supervised Models Shine: Self-supervised speech encoders (PASE+ and wav2vec 2.0) proved to be highly effective for perceptual loss training, suggesting their learned representations are robust and beneficial for denoising.
- Acoustic Event Classification is Key: The acoustic event classifier, trained on AudioSet, yielded the best performance when used individually with the l₁ loss (PERL-AE), highlighting the importance of preserving fundamental acoustic characteristics.
- l₁ Loss Remains Crucial: The simple STFT-domain l₁ loss is surprisingly resilient and effective, forming a strong baseline and complementing perceptual losses well.
- Domain Mismatch is a Real Challenge: When using all seven perceptual losses, the framework suffers from Multi-Task Learning challenges, particularly domain mismatch between auxiliary tasks and the denoising task.
- Hand-Tuning Beats Automatic Weighting: State-of-the-art automatic MTL weight learning methods did not outperform carefully hand-tuned weights, suggesting that current MTL techniques struggle with the specific challenges of this ensemble of perceptual losses.
Key Results
- The PERL framework, particularly the PERL-AE model (using only the acoustic event classifier and l₁ loss), outperforms state-of-the-art methods on major perceptual metrics (PESQ, CSIG, CBAK, COVL, STOI) on the VCTK-DEMAND dataset. For instance, PERL-AE achieves a PESQ score of 3.17 and STOI of 95.0, compared to a baseline l₁ loss with Conformer achieving 3.01 and 94.9 respectively.
- Using the acoustic event classifier alone with the l₁ loss (PERL-AE) achieved a PESQ score of 3.17, a significant improvement over previous work using deep feature loss for acoustic event classification (PESQ 2.57).
- Combining the l₁ loss with individual perceptual losses generally improved performance over the baseline l₁ loss.
- When using all seven losses (the full PERL framework), performance degraded compared to the baseline, indicating challenges with the ensemble and weighting.
- Hand-tuned weights for the PERL system (l₁+AcousticEvent) achieved the best results, with a PESQ of 3.17 and STOI of 95.0.
Limitations and Future Work
- Domain Mismatch: The authors acknowledge that domain mismatch between the pre-trained auxiliary networks and the denoising task is a significant challenge. They suggest future work could involve learning "bridges" or "adaptors" between these networks.
- MTL Weighting: Current automatic MTL weight learning methods are not effective for this specific ensemble of losses. Future work could explore more advanced MTL techniques or focus on better data augmentation and domain adaptation strategies.
- Evaluation: The current evaluation is restricted to objective enhancement metrics due to the lack of labels for all auxiliary tasks in the test set.
- Future Directions:
- Building a universal enhancement system by analyzing PERL on richer test setups.
- Incorporating automatic speech quality quantification networks like SESQA.
- Learning adaptors to alleviate domain mismatch.
- Exploring time-domain models (like DEMUCS) for phase prediction to further improve results.
Summary updated