The NPU-Elevoc Personalized Speech Enhancement System for ICASSP2023 DNS Challenge

Xiaopeng Yan, Yindi Yang, Zhihao Guo, Liangliang Peng, Lei XieICASSP 2023

TL;DR

The NPU-Elevoc team developed NAPSE, a high-performance personalized speech enhancement system that secured top honors at the ICASSP 2023 DNS Challenge. By combining a two-stage neural architecture with a novel speaker-embedding fusion strategy and adversarial training (MetricGAN), the system effectively isolates a target speaker's voice from complex noise and interference in real-time.

What Problem Does This Paper Address?

In a crowded room or a noisy office, human listeners are remarkably good at focusing on a single voice—a phenomenon known as the "cocktail party effect." Replicating this in software is called Personalized Speech Enhancement (PSE).

Unlike standard noise suppression, PSE uses a short "enrollment" clip (a reference of the target person's voice) to extract that specific speaker while filtering out background noise, reverberation, and even other people talking. The 5th Deep Noise Suppression (DNS) Challenge pushed the boundaries of this field by requiring systems to handle full-band audio (48kHz) across two scenarios: headset microphones and speakerphones.

Methods

The NAPSE system is built on three main pillars:

  1. Two-Stage Enhancement (TEA-PSE 2.0): The model breaks the problem into two parts. First, MAG-Net handles the coarse magnitude of the speech. Second, COM-Net refines the complex features (both magnitude and phase) to produce a natural-sounding result.
  2. Hybrid Speaker Encoding: To "know" who to listen for, the system uses a powerful ResNet34 encoder. Crucially, the authors found that combining these high-level neural embeddings with simple, low-level acoustic features (Fbank mean and standard deviation) through a learnable fusion module significantly improved the system's ability to generalize to new speakers.
  3. Adversarial Training (MetricGAN): Standard loss functions (like Mean Squared Error) don't always correlate with how humans perceive sound quality. The authors integrated MetricGAN, which uses a "discriminator" network to estimate perceptual scores like PESQ and DNSMOS. The enhancement model then uses these estimates as feedback to optimize for actual human-perceived quality.

The training followed a specific three-step pipeline: train the first stage, freeze it to train the second stage, and finally perform joint fine-tuning of the entire system.

Datasets

The researchers utilized a massive amount of data to ensure robustness:

  • DNS4 & DNS5 Datasets: Official challenge data containing thousands of hours of speech and noise.
  • Simulated Environments: They generated 150,000 Room Impulse Response (RIR) clips to simulate different acoustic spaces with varying reverberation times (0.1s to 1.2s).
  • Total Volume: Approximately 2,000 hours of simulated data were used for final training.

Key Insights and Ideas

  • The Power of Fusion: One of the "aha" moments was discovering that deep neural embeddings alone aren't perfect. Adding simple filterbank (Fbank) statistics helped the model better preserve the integrity of the speaker's identity across different environments.
  • Staged Training is Essential: The authors proved that the order of training matters. Jointly training the two stages at the end provided a significant performance boost over training them in isolation.
  • Optimizing for the Human Ear: By using MetricGAN to predict non-differentiable metrics (like DNSMOS), the model learned to prioritize speech naturalness and background suppression in a way that traditional mathematical losses cannot.

Key Results

The NAPSE system demonstrated state-of-the-art performance in the DNS Challenge:

  • Track 1 (Headset): Tied for 1st place. It achieved a final score of 0.590 (compared to the noisy baseline of 0.449).
  • Track 2 (Speakerphone): Ranked 2nd place with a score of 0.581.
  • Efficiency: Despite its complexity, the model is efficient. It has 12.49 million parameters and an Average Real-Time Factor (RTF) of 0.48 on a standard Intel Xeon CPU, meaning it can process audio twice as fast as it is recorded.

Limitations and Future Work

While the paper demonstrates a winning architecture, a few areas remain for exploration:

  • Model Complexity: While it runs in real-time on a server-grade CPU, the 12.49M parameter count might still be heavy for low-power mobile or IoT devices.
  • Metric Estimation: The system relies on MetricGAN to estimate perceptual scores. If the discriminator's estimates are inaccurate, it could lead the generator astray. Future work might involve even more robust perceptual loss functions.

Not discussed in the paper are the specific failure cases—such as when the interfering speaker sounds very similar to the target speaker—which remains a classic challenge in the PSE field.

Summary updated

Abstract

This paper describes our NPU-Elevoc personalized speech enhancement system (NAPSE) for the 5th Deep Noise Suppression Challenge at ICASSP 2023. Based on the superior two-stage model TEA-PSE 2.0, our system particularly explores better strategy for speaker embedding fusion, optimizes the model training pipeline, and leverages adversarial training and multi-scale loss. According to the results, our system is tied for the 1st place in the headset track (track 1) and ranked 2nd in the speakerphone track (track 2).
BibTeX
@inproceedings{yan2023npu,
  author = {Xiaopeng Yan and Yindi Yang and Zhihao Guo and Liangliang Peng and Lei Xie},
  title = {{The NPU-Elevoc Personalized Speech Enhancement System for ICASSP2023 DNS Challenge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
}
Figure