Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation

Zhong-Qiu Wang, Gordon Wichern, Jonathan Le Roux

Abstract

This paper presents a hybrid approach for monaural speech dereverberation and speaker separation that bridges the gap between deep learning and classical signal processing. By integrating deep neural networks (DNNs) with a physical model of reverberation via Forward Convolutive Prediction (FCP), the proposed method explicitly accounts for the linear convolution process of room acoustics. Unlike traditional methods such as Weighted Prediction Error (WPE), FCP effectively targets early reflections without requiring a prediction delay. Experimental results on the REVERB, SMS-WSJ, and WHAMR! datasets demonstrate state-of-the-art performance, with the monaural system outperforming multi-microphone baselines in complex acoustic environments.

Introduction

Reverberation—the collection of delayed and decayed sound reflections in a room—poses a significant challenge for automated speech processing systems. While humans can often filter out these echoes, they severely degrade the accuracy of transcription services and voice assistants.

Existing solutions typically fall into two categories: classical signal processing and black-box deep learning. Classical methods like WPE use mathematical models to subtract late echoes but struggle with early reflections (within 50ms) because they lack a prior model of clean speech. Conversely, purely data-driven DNNs map noisy audio to clean audio but often ignore the physical reality that reverberation is a linear convolution process. This paper introduces a system that utilizes DNNs to inform a physical model, capturing structural cues that purely statistical models might miss.

Proposed Method: Forward Convolutive Prediction (FCP)

The core innovation is the Forward Convolutive Prediction (FCP) framework, which operates in three stages to refine speech estimates iteratively.

  1. Initial Estimation (DNN1): A neural network produces a first-pass estimate of the direct-path (clean) speech from the reverberant mixture.
  2. Physical Modeling (FCP): The system solves a linear regression problem to determine the room impulse response (RIR) that would transform the DNN1 estimate into the observed reverberant signal. This step identifies the specific echoes associated with the target speaker.
  3. Refinement (DNN2): A second neural network takes the original mixture, the initial guess, and the identified echoes as inputs. By explicitly "knowing" which components are echoes, DNN2 can produce a significantly cleaner final output.
The proposed iterative framework for speech dereverberation using Forward Convolutive Prediction (FCP).

Architectural Details

The DNNs utilize a complex-valued encoder-decoder architecture optimized for Time-Frequency domain processing. This approach, known as Complex Spectral Mapping, allows the model to handle both magnitude and phase, which is critical for natural-sounding speech reconstruction. The bottleneck of the architecture features a 4-layer Temporal Convolutional Network (TCN) to capture long-range temporal dependencies in the speech signal.

Detailed architecture of the DNNs, featuring an encoder-decoder structure with a 4-layer Temporal Convolutional Network (TCN).

Experimental Results

The system was evaluated on three major benchmarks: REVERB (dereverberation), SMS-WSJ (two-speaker separation), and WHAMR! (noisy-reverberant separation).

  • Dereverberation Performance: On the REVERB dataset, the monaural system achieved a 7.9% Word Error Rate (WER), notably outperforming a standard 8-microphone beamforming baseline (8.8% WER).
  • Speaker Separation: On the WHAMR! dataset, which includes non-stationary background noise, the model achieved an SI-SDR of 7.5 dB, surpassing previous state-of-the-art models like Wavesplit.
  • Early Reflection Removal: Qualitative analysis confirmed that FCP is significantly more effective at removing early reflections than the industry-standard WPE algorithm, largely due to its lack of a required prediction delay.

Key Insights and Future Work

A critical finding was the advantage of the "per speaker" setup in multi-speaker scenarios. Running the second DNN separately for each speaker proved more effective than joint enhancement, as each speaker has a unique physical relationship (and thus a unique filter) with the microphone.

Future research directions include reducing the computational overhead of the per-speaker approach and implementing end-to-end training by backpropagating through the linear regression block. Additionally, adapting the model to handle time-variant filters for moving speakers remains an open challenge.

Summary updated

Abstract

A promising approach for speech dereverberation is based on supervised learning, where a deep neural network (DNN) is trained to predict the direct sound from noisy-reverberant speech. This data-driven approach is based on leveraging prior knowledge of clean speech patterns, and seldom explicitly exploits the linear-filter structure in reverberation, i.e., that reverberation results from a linear convolution between a room impulse response (RIR) and a dry source signal. In this work, we propose to exploit this linear-filter structure within a deep learning based monaural speech dereverberation framework. The key idea is to first estimate the direct-path signal of the target speaker using a DNN and then identify signals that are decayed and delayed copies of the estimated direct-path signal, as these can be reliably considered as reverberation. They can be either directly removed for dereverberation, or used as extra features for another DNN to perform better dereverberation. To identify the copies, we estimate the underlying filter (or RIR) by efficiently solving a linear regression problem per frequency in the time-frequency domain. We then modify the proposed algorithm for speaker separation in reverberant and noisy-reverberant conditions. State-of-the-art speech dereverberation and speaker separation results are obtained on the REVERB, SMS-WSJ, and WHAMR! datasets.
BibTeX
@article{wang2021convolutive,
  author = {Zhong-Qiu Wang and Gordon Wichern and Jonathan Le Roux},
  title = {{Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=HHR9i-Eol2d},
}
Figure