TL;DR
This paper introduces TorchAudio-Squim, a new toolkit within PyTorch's TorchAudio library that provides reference-less deep learning models for estimating speech quality and intelligibility. By offering easy-to-use, non-intrusive models for metrics like PESQ, STOI, SI-SDR, and MOS, it aims to significantly benefit speech processing research and development by removing the need for clean reference audio.
What Problem Does This Paper Address?
Evaluating the quality and intelligibility of speech signals is crucial for developing and assessing speech processing systems. Traditionally, this has been done using subjective listening tests (like Mean Opinion Score - MOS) or objective metrics. Subjective tests are reliable but time-consuming and not scalable. Objective metrics, while faster, often require a clean "reference" speech signal to compare against the degraded signal. This "intrusive" nature makes them impractical for real-world applications where clean references are unavailable. While reference-less methods have emerged, their integration into existing workflows has been hindered by a lack of accessible tools. This paper addresses this gap by providing readily usable, deep learning-based, reference-less models for speech quality and intelligibility assessment within a popular audio processing library.
Methods
The core of TorchAudio-Squim lies in its deep learning models. For objective metrics (PESQ, STOI, SI-SDR), the authors propose a novel approach using a dual-path recurrent neural network (DPRNN) architecture. This network processes time-domain speech signals and uses multi-head attention and transformer blocks to estimate all three metrics simultaneously. A key aspect is the multi-task learning (MTL) strategy, where the model is trained not only to predict the metric scores but also to reconstruct the reference signal. This secondary task is hypothesized to help the shared layers learn better representations of the speech signal, thereby improving metric estimation.
For subjective metrics, specifically Mean Opinion Score (MOS), they leverage the NORESQA-MOS model. This approach, while still requiring a clean speech sample, uses a "non-matching reference" (NMR) strategy. This means any clean speech sample can be used as a reference, making it more flexible than methods requiring a direct match to the degraded signal. The models are integrated into TorchAudio's torchaudio.models and torchaudio.pipelines modules for easy access and use.
Datasets
The primary dataset used for training and evaluation in this work is the DNS Challenge 2020 dataset. Degraded speech samples were generated by mixing clean speech with additive noise across various Signal-to-Noise Ratios (SNRs) and by processing some noisy mixtures through speech enhancement systems. The authors mention that the training, validation, and test sets consist of approximately 364,500, 14,600, and 22,800 audio samples, respectively. All training signals were truncated to 5 seconds.
Key Insights and Ideas
- Reference-less is the way forward: The paper strongly advocates for and demonstrates the effectiveness of reference-less methods, highlighting their practical utility in real-world scenarios where clean audio is often unavailable.
- Multi-task learning for objective metrics: Training a single model to predict multiple objective metrics and reconstruct the reference signal proved beneficial. This suggests that the auxiliary task of reference reconstruction helps the model learn more robust and generalizable features for quality and intelligibility estimation.
- Unified model for multiple metrics: The proposed DPRNN-based architecture efficiently estimates STOI, PESQ, and SI-SDR simultaneously. This is more computationally efficient and potentially leads to better performance than training separate models for each metric.
- Accessibility is key: By integrating these advanced models into the widely-used TorchAudio library, the authors aim to democratize access to state-of-the-art speech assessment tools, fostering further research and development.
Key Results
The proposed TorchAudio-Squim system, particularly with multi-task learning (MTL), demonstrated superior performance compared to several existing reference-less speech assessment models.
- Objective Metrics: The MTL-enabled model achieved the best results across STOI, WB-PESQ, and SI-SDR metrics, showing lower Mean Absolute Error (MAE) and higher Pearson (PCC) and Spearman (SRCC) correlation coefficients. For instance, using MTL, the MAE for WB-PESQ improved from 0.168 to 0.142, and PCC/SRCC increased from 0.942/0.951 to 0.958/0.963.
- Comparison to Baselines: The paper shows that their model significantly outperforms established methods like Quality-Net, MOSA-Net, AMSA, and MetricNet in terms of MAE, PCC, and SRCC.
- Multi-objective learning benefits: Training a single model for multiple objectives not only improved performance but also offered computational efficiency compared to training individual models.
Limitations and Future Work
- Subjective Metric Reference: While the objective metrics are truly reference-less, the subjective MOS estimation (NORESQA-MOS) still requires a clean speech sample, albeit a non-matching one. This is a limitation for scenarios where any clean reference is unavailable.
- Model Complexity: The DPRNN architecture, while effective, can be computationally intensive. The authors mention exploring more computationally efficient models in future work.
- Future Directions: The authors plan to continuously develop TorchAudio-Squim by:
- Improving existing models for better metric estimation.
- Reducing computational load.
- Extending the system to estimate more objective and subjective speech metrics.
- Exploring more efficient model architectures.
Summary updated