A Deep Learning Loss Function based on Auditory Power Compression for Speech Enhancement

Tianrui WangarXiv 2021 · arXiv

TL;DR

This paper introduces a novel loss function, Auditory Power Compression Signal-to-Noise Ratio (APC-SNR), for deep learning-based speech enhancement. By incorporating auditory characteristics and power compression, APC-SNR demonstrates improved correlation with objective speech intelligibility metrics and leads to better training performance and overall speech quality compared to existing methods.

What Problem Does This Paper Address?

Speech enhancement aims to improve the clarity and quality of speech signals, often corrupted by noise. While deep learning has made significant strides, the effectiveness of the loss function used to train these models remains a critical area for improvement. Existing loss functions, such as Mean Squared Error (MSE), Scale-Invariant SNR (SI-SNR), and Perceptual Evaluation of Speech Quality (PESQ)-based metrics, often struggle with accurately reflecting human auditory perception and achieving strong correlations with subjective intelligibility scores. Specifically, researchers are looking for loss functions that better capture the nuances of human hearing, are more robust to variations in signal characteristics, and provide a more effective gradient for training deep learning models.

Methods

The core contribution of this paper is the Auditory Power Compression Signal-to-Noise Ratio (APC-SNR) loss function. This method builds upon the concept of auditory perception by:

  • Auditory Power Compression: It simulates the non-linear compression of sound intensity as perceived by the human ear. This is achieved by mapping the power spectrum to a compressed form, inspired by loudness perception models like Zwicker's.
  • Time-Frequency (T-F) Domain Adaptation: The loss function is formulated in the T-F domain, allowing it to consider both magnitude and phase information, similar to SI-SNR.
  • Magnitude and Phase Sensitivity: Unlike some T-F domain methods that can be phase-insensitive, APC-SNR aims to provide more controllable focus on both magnitude and phase.
  • Gradient Improvement: The authors address the "gradient blur" issue observed in some existing methods by simplifying the auditory loudness formula and introducing a limiting threshold for scaling factors, ensuring a more stable and effective gradient during training.

The paper also discusses the Scale-Invariant Signal-to-Noise Ratio in the T-F spectrum (SI-SNRTF) as a baseline T-F domain loss, highlighting its limitations in phase sensitivity.

Datasets

  • DNS-Challenge [17]: Used to generate approximately 200 hours of noisy speech for training and validation, with SNRs ranging from -5dB to 20dB and pitch shifts between -2 and 2 semitones.
  • ESC-50 [19]: Used for noise sources in the test set.
  • Custom Test Set: Approximately 38 hours of audio generated for testing, with SNRs from -10dB to 30dB and pitch shifts between -2 and 2 semitones. The test set speech did not overlap with the training/validation data.

Key Insights and Ideas

  • Auditory Compression is Key: Directly incorporating auditory power compression into the loss function significantly improves its correlation with human perception and intelligibility metrics.
  • T-F Domain Benefits: While time-domain SI-SNR is effective, extending it to the T-F domain (SI-SNRTF) is necessary for more comprehensive signal representation, but requires careful handling of phase sensitivity.
  • Gradient Stability Matters: Issues like "gradient blur" in existing methods hinder effective training. The proposed APC-SNR addresses this by simplifying auditory models and introducing limiting mechanisms.
  • Combined Loss is Powerful: Combining APC-SNR with existing metrics like PMSQE1 (a PESQ-based metric) yielded the best overall performance, suggesting that APC-SNR can compensate for the weaknesses of other loss functions.

Key Results

  • Improved Correlation: The proposed APC-SNR loss function showed a better overall correlation with objective speech intelligibility indexes (like PESQ and STOI) compared to MSE, PMSQE, and SI-SNR.
  • Enhanced Training Performance: Models trained with APC-SNR exhibited better training effects and achieved a higher comprehensive index (CI) of 0.570 compared to other individual loss functions.
  • State-of-the-Art Performance: When combined with PMSQE1, the APC-SNR achieved the highest CI of 0.838, outperforming other tested combinations.
  • Hyperparameter Sensitivity: The paper analyzed the impact of hyperparameters $\theta$ and $\epsilon$ in the APC-SNR formula, finding optimal values of $\epsilon=0.01$ and $\theta=1$ for balancing compression and preventing overfitting.

Limitations and Future Work

  • Hyperparameter Tuning: While optimal values for $\theta$ and $\epsilon$ were found, their sensitivity suggests that further research into adaptive or automatic hyperparameter selection could be beneficial.
  • Further Combination Exploration: The success of combining APC-SNR with PMSQE1 suggests that exploring other combinations of APC-SNR with various loss functions could lead to even better performance.
  • Subjective Evaluation: While the paper relies on objective metrics and correlation analysis, direct subjective listening tests would provide a more definitive evaluation of the perceived speech quality improvements.

Summary updated

BibTeX
@misc{wang2021deep,
  author = {Tianrui Wang},
  title = {{A Deep Learning Loss Function based on Auditory Power Compression for Speech Enhancement}},
  year = {2021},
  eprint = {2108.11877},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.11877},
}
Figure