Icassp 2022 Deep Noise Suppression Challenge

Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Sefik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, Robert Aichner

TL;DR

The 4th Deep Noise Suppression (DNS) Challenge pushes the boundaries of speech enhancement by moving to 48 kHz fullband audio and introducing Personalized DNS, which suppresses not just background noise but also neighboring talkers. By incorporating Word Accuracy (WAcc) as a primary metric, the challenge ensures that denoising models improve human perception without sacrificing machine intelligibility.

What Problem Does This Paper Address?

In the era of hybrid work, background noise—from barking dogs to kitchen clatter—has become a major source of "meeting fatigue." While deep learning-based noise suppression has made massive strides, two major gaps remained:

  1. The "Other Voice" Problem: Standard noise suppression treats all human speech as a signal to be preserved. In a shared office or home, this means your colleague's or partner's voice is amplified along with yours.
  2. The Intelligibility Trade-off: Aggressive denoising often introduces "artifacts" or over-suppression that makes speech sound robotic or muffled, which can break Automatic Speech Recognition (ASR) systems.
  3. Audio Quality: Most previous research focused on 16 kHz (wideband) audio, whereas modern communication platforms are moving toward 48 kHz (fullband) for high-fidelity "crystal clear" sound.

Methods

The challenge was split into two tracks:

  • Track 1 (Non-personalized DNS): Traditional suppression of non-speech background noise for fullband audio.
  • Track 2 (Personalized DNS/PDNS): Suppressing both background noise and "interfering" talkers. This requires the model to know who the "primary" speaker is.

The "Secret Sauce" for Personalization: To achieve personalization, participants were given 2.5 minutes of "enrollment speech" for the primary user. This allowed models to extract a speaker embedding (using a baseline RawNet2 model). During inference, the model uses this embedding as a reference to "lock onto" the primary voice and filter out everyone else.

Evaluation Innovation: The organizers moved beyond simple subjective scores. They used:

  • ITU-T P.835: A subjective framework that gives three separate scores: Speech Quality (SIG), Background Noise Quality (BAK), and Overall Quality (OVRL).
  • WAcc (Word Accuracy): An objective metric using a state-of-the-art ASR system to ensure the denoised audio is still "readable" by machines.
  • DNSMOS P.835: A neural network-based objective metric that mimics human P.835 ratings, allowing researchers to iterate quickly without expensive human listening tests.

Datasets

The challenge provided a massive open-source repository:

  • Clean Speech: Over 3,200 talkers across six languages (English, French, German, Italian, Russian, Spanish), including a unique high-quality singing dataset (VocalSet).
  • Noise: ~62,000 clips across 150 classes (fans, typing, sirens, etc.) sourced from AudioSet and Freesound.
  • Impulse Responses: ~60,000 synthetic and 248 real room impulse responses (RIRs) to simulate realistic reverberation.
  • Blind Test Set: A new set of real-world recordings, 70% of which were captured on mobile devices to reflect modern usage patterns.

Key Insights and Ideas

  • Personalization is the New Frontier: The PDNS track showed that speaker-aware models can significantly outperform general models in "crowded" environments.
  • The Mobile Advantage: Interestingly, MOS scores for clips recorded on mobile devices were higher than those from desktops. This suggests that modern smartphone microphone arrays and hardware-level processing provide a better starting point for software-based DNS.
  • The "Aggression" Trap: The organizers noted that many models in previous years achieved high noise suppression but "mangled" the speech so much that Word Accuracy plummeted. By making WAcc part of the final ranking score, they forced a balance between silence and clarity.
  • DNSMOS is Reliable: The high correlation (0.92–0.96 PCC) between the DNSMOS model and human raters validates it as a powerful tool for the community to use for rapid prototyping.

Key Results

  • Participation: 24 teams submitted to Track 1, and 10 teams tackled the more difficult Track 2.
  • Efficiency vs. Performance: The winning model for Track 1 (Team 2) was remarkably efficient, using only 1.5M parameters with a real-time factor of 0.60.
  • The "Room for Improvement": A sobering finding was that while models significantly improved background noise (BAK), none of the submissions actually improved the raw Speech Quality (SIG) or Word Accuracy (WAcc) compared to the original noisy clips. Most models "broke even" on speech quality while cleaning up the background.
  • Track 2 Success: The winning PDNS model (Team 42) achieved a significantly higher overall score than the baseline, proving that personalized denoising is ready for prime time.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Diversity: Future challenges need a wider variety of languages and accents to ensure global fairness.
  • Standardized Benchmarking: While they provided a baseline, they plan to implement a dedicated inference engine setup to more accurately measure model complexity and latency across different architectures.
  • Look-ahead Constraints: To ensure fairness in real-time applications, stricter validation of "look-ahead" (algorithmic delay) will be required in future iterations.

Summary updated

Abstract

The Deep Noise Suppression (DNS) challenge is designed to foster innovation in the area of noise suppression to achieve superior perceptual speech quality. This is the 4th DNS challenge, with the previous editions held at INTERSPEECH 2020 [1], ICASSP 2021 [2], and INTERSPEECH 2021 [3]. We open-source datasets and test sets for researchers to train their deep noise suppression models, as well as a subjective evaluation framework based on ITU-T P.835 to rate and rank-order the challenge entries. We provide access to DNS-MOS P.835 and word accuracy (WAcc) APIs to challenge participants to help with iterative model improvements. In this challenge, we introduced the following changes: (i) Included mobile device scenarios in the blind test set; (ii) Included a personalized noise suppression track with baseline; (iii) Added WAcc as an objective metric; (iv) Included DNSMOS P.835; (v) Made the training datasets and test sets fullband (48 kHz). We use an average of WAcc and subjective scores P.835 SIG, BAK, and OVRL to get the final score for ranking the DNS models. We believe that as a research community, we still have a long way to go in achieving excellent speech quality in challenging noisy real-world scenarios.
BibTeX
@inproceedings{dubey2022icassp,
  author = {Harishchandra Dubey and Vishak Gopal and Ross Cutler and Ashkan Aazami and Sergiy Matusevych and Sebastian Braun and Sefik Emre Eskimez and Manthan Thakker and Takuya Yoshioka and Hannes Gamper and Robert Aichner},
  title = {{Icassp 2022 Deep Noise Suppression Challenge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=BJask8-CZ8_},
}

Loading related content...

Figure