TEA-PSE 3.0: Tencent-Ethereal-Audio-Lab Personalized Speech Enhancement System For ICASSP 2023 Dns-Challenge

Yukai Ju, Jun Chen, Shimin Zhang, Shulin He, Wei Rao, Weixin Zhu, Yannan Wang, Tao Yu, Shidong ShangICASSP 2023

TL;DR

The Tencent Ethereal Audio Lab team has developed TEA-PSE 3.0, a personalized speech enhancement system that achieved first place in the ICASSP 2023 DNS Challenge. Their approach builds on previous work by incorporating a residual LSTM for improved sequence modeling, a local-global representation (LGR) for better speaker information extraction, and a multi-STFT resolution loss for enhanced time-frequency feature capture.

What Problem Does This Paper Address?

The core problem this paper tackles is speech enhancement, specifically personalized speech enhancement. In noisy environments, speech signals are often corrupted, making them difficult to understand. Traditional noise suppression methods can be generic, but personalized systems aim to enhance speech for a specific target speaker, leading to more natural and intelligible output. The challenge is to effectively separate the target speaker's voice from background noise while preserving the speaker's unique characteristics and the speech's temporal and spectral nuances. This is particularly important for applications like voice assistants, teleconferencing, and hearing aids, where clear communication is paramount.

Methods

TEA-PSE 3.0 is a two-stage framework consisting of a MAG-Net (for magnitude) and a COM-Net (for complex-valued features). Key innovations include:

  • Residual LSTM after S-TCN: To improve sequence modeling capabilities, a residual LSTM layer is added after each Squeezed Temporal Convolutional Network (S-TCN) module. This helps the model better capture long-range dependencies in the speech signal.
  • Local-Global Representation (LGR): This structure is introduced to enhance the extraction of speaker-specific information. It combines local and global features from an "anchor" speech sample to better identify and isolate the target speaker's voice.
  • Multi-STFT Resolution Loss: Instead of using a single Short-Time Fourier Transform (STFT) configuration, the system employs multiple STFT resolutions. This allows the model to capture speech signal characteristics across different time-frequency granularities, leading to more robust enhancement.
  • Freeze Training Strategy: A three-step training strategy is used. First, the stage-one model (MAG-Net) is trained and frozen. Then, the stage-two model (COM-Net) is trained using the frozen stage-one model. Finally, the entire system is fine-tuned with the second stage's loss function.

Datasets

  • Training and Evaluation: The ICASSP 2022 DNS-challenge full-band dataset is used.
  • Noise Data: DEMAND, Freesound, and AudioSet datasets are utilized for noise generation.
  • Room Impulse Responses (RIRs): 100,000 RIRs were generated using the image method to simulate various acoustic environments.

Key Insights and Ideas

  • Sequence Modeling is Crucial: The addition of LSTMs after S-TCN layers significantly boosts performance, highlighting the importance of capturing temporal dynamics in speech enhancement.
  • Speaker Identity Matters: The LGR structure proves effective in leveraging speaker-specific information, leading to better target speaker extraction and cleaner output.
  • Multi-Resolution Analysis Pays Off: Using multiple STFT resolutions provides a more comprehensive view of the speech signal's time-frequency characteristics, improving overall enhancement quality.
  • Strategic Training is Key: The freeze training strategy, combined with retraining, offers an additional performance gain, suggesting that a phased approach to model training can be highly beneficial.

Key Results

TEA-PSE 3.0 achieved 1st place in both the headset and non-headset tracks of the ICASSP 2023 DNS Challenge. Specifically, the paper reports:

  • PDNSMOS P.835 Results (Table 1): The final model (including +LSTM, +LGR, +multi-STFT, and +retrain) achieved the highest scores across various metrics (SIG, BAK, OVRL) for both Track 1 and Track 2 compared to baseline and intermediate versions of their model. For example, in Track 1, the final model achieved SIG 4.108, BAK 4.053, and OVRL 3.645.
  • MOS and WAcc Results (Table 2): TEA-PSE 3.0 achieved the highest BAK and OVRL scores on the DNS 2023 blind test set. While SIG and WAcc slightly decreased compared to unprocessed speech, this is noted as a reasonable trade-off for the introduced distortion reduction.
  • Efficiency: The model has 22.24 million trainable parameters and an average real-time factor (RTF) of 0.46 on an Intel Xeon CPU.

Limitations and Future Work

  • Slight Distortion: The paper acknowledges that the enhancement process, while improving intelligibility and noise reduction, can lead to a slight decrease in the Signal (SIG) and Word Accuracy (WAcc) metrics compared to unprocessed speech. This is a common trade-off in speech enhancement.
  • Future Work: While not explicitly detailed, the success of TEA-PSE 3.0 suggests that further exploration of advanced sequence modeling techniques, more sophisticated speaker embedding methods, and novel multi-resolution analysis strategies could lead to even more robust and natural speech enhancement systems. The current work also sets a strong baseline for future personalized speech enhancement research.

Summary updated

BibTeX
@inproceedings{ju2023tea,
  author = {Yukai Ju and Jun Chen and Shimin Zhang and Shulin He and Wei Rao and Weixin Zhu and Yannan Wang and Tao Yu and Shidong Shang},
  title = {{TEA-PSE 3.0: Tencent-Ethereal-Audio-Lab Personalized Speech Enhancement System For ICASSP 2023 Dns-Challenge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
}
Figure