Abstract
The INTERSPEECH 2020 Deep Noise Suppression (DNS) Challenge is intended to promote collaborative research in real-time single-channel Speech Enhancement aimed to maximize the subjective (perceptual) quality of the enhanced speech. A typical approach to evaluate the noise suppression methods is to use objective metrics on the test set obtained by splitting the original dataset. While the performance is good on the synthetic test set, often the model performance degrades significantly on real recordings. Also, most of the conventional objective metrics do not correlate well with subjective tests and lab subjective tests are not scalable for a large test set. In this challenge, we open-sourced a large clean speech and noise corpus for training the noise suppression models and a representative test set to real-world scenarios consisting of both synthetic and real recordings. We also open-sourced an online subjective test framework based on ITU-T P.808 for researchers to reliably test their developments. We evaluated the results using P.808 on a blind test set. The results and the key learnings from the challenge are discussed.
Introduction: Bridging the "Reality Gap"
For years, the speech enhancement (SE) community has relied on objective metrics like PESQ (Perceptual Evaluation of Speech Quality) and SDR (Speech to Distortion Ratio) to evaluate models. However, these metrics often fail to correlate with how humans actually perceive audio quality. Furthermore, models that perform brilliantly on synthetic test sets—where clean speech and noise are mixed mathematically—frequently "break" when faced with real-world recordings involving complex reverberation and non-stationary noise.
The INTERSPEECH 2020 Deep Noise Suppression (DNS) Challenge addresses this "reality gap" by providing a massive, open-source dataset and a scalable, crowdsourced subjective testing framework based on ITU-T P.808. By moving beyond unreliable objective metrics, the challenge established a new benchmark for real-world perceptual quality in both real-time and non-real-time audio processing.
Methodology and Evaluation Framework
The challenge was structured into two tracks to address different computational constraints:
- Track 1 (Real-Time): Focused on low-latency applications. Algorithms had to process a frame in less than half the frame's duration on a standard laptop CPU, with a maximum look-ahead of 40ms.
- Track 2 (Non-Real-Time): Allowed for deeper, more computationally expensive models to push the boundaries of absolute audio quality.
To evaluate these, the authors utilized Amazon Mechanical Turk (MTurk) to crowdsource Mean Opinion Scores (MOS). To ensure data integrity, they implemented rigorous quality controls, including hearing tests for raters, "gold" clips with known quality, and "trapping questions" to filter out inattentive participants. This framework proved that crowdsourcing can match the accuracy of expensive lab tests, achieving a 0.90 Spearman correlation with traditional methods.
Dataset Curation
The authors provided a massive upgrade over previous benchmarks, focusing on high-quality source material and diversity:
- Clean Speech: ~500 hours of audio from 2,150 speakers, derived from Librivox audiobooks. Crucially, they used their P.808 framework to filter the audio, keeping only the top 25% of clips (MOS ≥ 4.3) to ensure a truly "clean" reference.
- Noise Dataset: ~60,000 clips across 150 classes from Audioset and Freesound, balanced to ensure rare but important noises weren't overshadowed by common ones.
- Test Sets: A mix of synthetic clips (with and without reverb) and real-world recordings collected internally at Microsoft and from AudioSet.
Key Results and Insights
The challenge received 28 submissions from 19 teams. The top-performing models significantly outperformed the baseline NSNet (a GRU-based recurrent neural network).
Key insights from the results include:
- The Reverb Challenge: Adding reverberation to synthetic training data is essential for real-world performance. Models trained only on "dry" speech fail when the speaker is in a large, echoey room.
- Complexity as a Tie-Breaker: In Track 2, where quality scores were often neck-and-neck, computational complexity was used as a secondary metric to reward efficiency.
- Statistical Significance: Using ANOVA, the researchers were able to clearly distinguish the top winners in the Real-Time track, though the Non-Real-Time track saw more overlap in performance.
Conclusion and Future Work
While the P.808 framework measures overall quality, it doesn't distinguish between speech distortion and residual noise. The authors suggest that future challenges should move toward ITU-T P.835, which provides separate scores for speech and background noise quality. Additionally, the team aims to explore personalized noise suppression and develop a "no-reference" MOS predictor—an AI model that can estimate subjective human scores without needing a human rater, drastically speeding up the development cycle for researchers.
Summary updated