Abstract
The Deep Noise Suppression (DNS) challenge is designed to foster innovation in the area of noise suppression to achieve superior perceptual speech quality. We recently organized a DNS challenge special session at INTERSPEECH 2020 where we open-sourced training and test datasets for researchers to train their noise suppression models. We also open-sourced a subjective evaluation framework and used the tool to evaluate and select the final winners. Many researchers from academia and industry made significant contributions to push the field forward. We also learned that as a research community, we still have a long way to go in achieving excellent speech quality in challenging noisy real-time conditions. In this challenge, we expanded both our training and test datasets. Clean speech in the training set has increased by 200% with the addition of singing voice, emotion data, and non-English languages. The test set has increased by 100% with the addition of singing, emotional, non-English (tonal and non-tonal) languages, and, personalized DNS test clips. There are two tracks with focus on (i) real-time denoising, and (ii) real-time personalized DNS. We present the challenge results at the end.
Introduction
As remote work and digital collaboration become ubiquitous, the demand for high-quality, real-time audio has skyrocketed. Background noises—such as barking dogs, crying babies, or kitchen clatter—significantly degrade speech intelligibility and cause listener fatigue. While noise suppression is a decades-old problem, traditional objective metrics (like PESQ or SDR) often fail to correlate with human perception in noisy, reverberant environments. Furthermore, many existing models struggle with "non-standard" speech like singing or high-emotion talk and often fail to meet the strict latency requirements needed for live communication. The ICASSP 2021 DNS Challenge aims to bridge these gaps by providing a standardized, perceptually-driven benchmark.
Methodology
The challenge emphasized real-time performance across two main tracks:
- Track 1: Real-Time Denoising. Models were required to process audio with a total algorithmic latency of $\le 40$ms on a standard quad-core processor, including frame size, stride, and look-ahead.
- Track 2: Personalized DNS (pDNS). This track introduced a more complex task where models were provided with 2 minutes of "clean" adaptation speech from a specific user. The goal was to leverage speaker-specific information to suppress both noise and neighboring speakers (crosstalk).
The "Gold Standard" for evaluation was subjective human testing based on the ITU-T P.808 framework. Crowdsourced human raters provided Mean Opinion Scores (MOS), ensuring results reflected true human preference rather than algorithmic approximations.
Dataset Expansion
To improve model generalization, the organizers significantly expanded the datasets from the previous year:
- Clean Speech: Increased to over 760 hours, incorporating read speech (Librivox), singing voices (VocalSet), emotional speech (CREMA-D), and a diverse array of non-English languages including Mandarin, Spanish, German, and French.
- Noise and RIRs: 181 hours of diverse noise events from Audioset and Freesound were utilized, alongside over 118,000 synthetic and real Room Impulse Responses (RIRs) to simulate realistic acoustics.
Key Results and Insights
Track 1: Real-Time Denoising
Performance varied significantly across different speech types. While top models performed well on standard English speech, singing and emotional speech proved much more difficult to enhance without distortion.
As shown in Table 1, the top-performing model (Team 24) achieved an overall Differential MOS (DMOS) of 0.53. However, the data reveals a "Singing and Emotion" gap; most models struggle with these categories because their pitch and intensity profiles are often mistaken for noise, leading to aggressive over-suppression.
Track 2: Personalized DNS
Personalization remains a significant challenge. This track saw lower participation and more modest gains compared to the general denoising track.
Table 2 illustrates that the best model in Track 2 achieved only a 0.14 DMOS improvement using speaker information. This suggests that leveraging speaker-specific data in a real-time, low-latency pipeline is a substantial architectural challenge that is still in its early stages of development.
Conclusion
The ICASSP 2021 DNS Challenge highlights both the progress and the remaining hurdles in speech enhancement. While the community is advancing toward better real-time models, a substantial gap remains between current state-of-the-art performance (MOS ~3.4) and the goal of "excellent" perceptual quality (MOS 4.0-5.0). Future work must focus on improving model robustness across diverse emotional states and refining personalized suppression techniques to solve the "cocktail party problem" in home environments.
Summary updated