Abstract
This paper presents a hybrid approach for monaural speech dereverberation and speaker separation that bridges the gap between deep learning and classical signal processing. By integrating deep neural networks (DNNs) with a physical model of reverberation via Forward Convolutive Prediction (FCP), the proposed method explicitly accounts for the linear convolution process of room acoustics. Unlike traditional methods such as Weighted Prediction Error (WPE), FCP effectively targets early reflections without requiring a prediction delay. Experimental results on the REVERB, SMS-WSJ, and WHAMR! datasets demonstrate state-of-the-art performance, with the monaural system outperforming multi-microphone baselines in complex acoustic environments.
Introduction
Reverberation—the collection of delayed and decayed sound reflections in a room—poses a significant challenge for automated speech processing systems. While humans can often filter out these echoes, they severely degrade the accuracy of transcription services and voice assistants.
Existing solutions typically fall into two categories: classical signal processing and black-box deep learning. Classical methods like WPE use mathematical models to subtract late echoes but struggle with early reflections (within 50ms) because they lack a prior model of clean speech. Conversely, purely data-driven DNNs map noisy audio to clean audio but often ignore the physical reality that reverberation is a linear convolution process. This paper introduces a system that utilizes DNNs to inform a physical model, capturing structural cues that purely statistical models might miss.
Proposed Method: Forward Convolutive Prediction (FCP)
The core innovation is the Forward Convolutive Prediction (FCP) framework, which operates in three stages to refine speech estimates iteratively.
- Initial Estimation (DNN1): A neural network produces a first-pass estimate of the direct-path (clean) speech from the reverberant mixture.
- Physical Modeling (FCP): The system solves a linear regression problem to determine the room impulse response (RIR) that would transform the DNN1 estimate into the observed reverberant signal. This step identifies the specific echoes associated with the target speaker.
- Refinement (DNN2): A second neural network takes the original mixture, the initial guess, and the identified echoes as inputs. By explicitly "knowing" which components are echoes, DNN2 can produce a significantly cleaner final output.
Architectural Details
The DNNs utilize a complex-valued encoder-decoder architecture optimized for Time-Frequency domain processing. This approach, known as Complex Spectral Mapping, allows the model to handle both magnitude and phase, which is critical for natural-sounding speech reconstruction. The bottleneck of the architecture features a 4-layer Temporal Convolutional Network (TCN) to capture long-range temporal dependencies in the speech signal.
Experimental Results
The system was evaluated on three major benchmarks: REVERB (dereverberation), SMS-WSJ (two-speaker separation), and WHAMR! (noisy-reverberant separation).
- Dereverberation Performance: On the REVERB dataset, the monaural system achieved a 7.9% Word Error Rate (WER), notably outperforming a standard 8-microphone beamforming baseline (8.8% WER).
- Speaker Separation: On the WHAMR! dataset, which includes non-stationary background noise, the model achieved an SI-SDR of 7.5 dB, surpassing previous state-of-the-art models like Wavesplit.
- Early Reflection Removal: Qualitative analysis confirmed that FCP is significantly more effective at removing early reflections than the industry-standard WPE algorithm, largely due to its lack of a required prediction delay.
Key Insights and Future Work
A critical finding was the advantage of the "per speaker" setup in multi-speaker scenarios. Running the second DNN separately for each speaker proved more effective than joint enhancement, as each speaker has a unique physical relationship (and thus a unique filter) with the microphone.
Future research directions include reducing the computational overhead of the per-speaker approach and implementing end-to-end training by backpropagating through the linear regression block. Additionally, adapting the model to handle time-variant filters for moving speakers remains an open challenge.
Summary updated