USDnet: Unsupervised Speech Dereverberation via Neural Forward Filtering

Zhong-Qiu Wang

This paper presents USDnet, a novel framework for unsupervised speech dereverberation. By combining the modeling power of deep neural networks (DNNs) with the physical constraints of multi-microphone recordings, the author demonstrates that we can train high-performing dereverberation models without ever seeing a single "clean" speech sample.

TL;DR

USDnet is an unsupervised deep learning approach that solves the "blind deconvolution" problem of speech dereverberation by using multi-microphone mixtures as physical constraints. It outperforms traditional signal-processing methods like WPE and generalizes better than supervised models because it can be trained directly on real-world reverberant recordings.

What Problem Does This Paper Address?

Room reverberation—the persistence of sound after a source has stopped—is a major hurdle for both human hearing and machine tasks like Automatic Speech Recognition (ASR).

Current solutions generally fall into two camps:

  1. Signal Processing (e.g., WPE): These are unsupervised but rely on relatively simple linear models that don't capture the complex spectro-temporal patterns of human speech.
  2. Supervised Deep Learning: These models are powerful but require pairs of "clean" and "reverberant" speech for training. Since we can't record "clean" speech in a reverberant room, researchers use simulations. Unfortunately, models trained on simulations often fail when they encounter the messy, unpredictable acoustics of the real world.

The challenge is: How do we give a DNN the "supervision" it needs to learn speech patterns without providing ground-truth clean labels?

Methods

The author treats dereverberation as a blind deconvolution problem. In a room with one speaker and multiple microphones (an "over-determined" system), the signal at each mic is just the source speech filtered by a different room impulse response.

USDnet uses a "Neural Forward Filtering" architecture:

  • The DNN: Takes a reverberant mixture and predicts an estimate of the anechoic (clean) speech.
  • Forward Convolutive Prediction (FCP): Instead of comparing the DNN output to a label, the system uses a differentiable linear regression layer to estimate the filters that would turn that "clean" estimate back into the original recorded mixtures at every microphone.
  • Mixture-Constraint (MC) Loss: The model is trained to minimize the difference between these reconstructed mixtures and the actual recorded ones.

To prevent the network from "cheating" (e.g., just outputting the input mixture), the author introduces several clever regularizations:

  • Positive Prediction Delay ($\Delta$): The filters are forced to ignore the most recent frames, ensuring the DNN handles the direct path and early reflections while the filters handle the late reverberation.
  • Complex Ratio Masking: The DNN predicts a mask to apply to the input rather than generating a signal from scratch. This keeps the output phase-aligned with the input.
  • Garbage Sources: A secondary output "absorbs" noise and modeling errors that don't fit the speech model.

Datasets

  • WSJ0CAM-DEREVERB: A simulated dataset based on the Wall Street Journal corpus, used to tune hyperparameters and provide a controlled environment for evaluation.
  • REVERB Challenge Dataset: A benchmark containing real-world recordings in various rooms (near-field and far-field). This was used to test how well the model generalizes to real data and its impact on ASR performance.

Key Insights and Ideas

  • Over-determination is a gift: Having multiple microphones provides extra "equations" for the same "unknown" (the source speech). This narrows down the infinite possible solutions of blind deconvolution.
  • Masking > Mapping: In unsupervised learning, "mapping" (direct signal generation) is too flexible. The network often finds "trivial solutions" where it outputs a time-shifted version of the input to satisfy the loss. Masking constrains the network's search space, forcing it to actually remove reverberation.
  • Training vs. Inference: Even though the model uses multiple microphones as a "constraint" during training, it can be configured to perform monaural (single-channel) dereverberation at test time.

Key Results

  • Better Quality: On the WSJ0CAM test set, USDnet achieved a PESQ score (speech quality) of 2.53, significantly higher than the standard WPE (1.78) and even a supervised DNN-WPE baseline (1.81).
  • ASR Gains: On the real-world REVERB dataset, USDnet reduced the Word Error Rate (WER) from 17.2% (unprocessed) to 10.4%. This outperformed WPE (14.2%) by a wide margin.
  • Visual Clarity: Spectrograms show that USDnet produces much sharper speech patterns compared to the "smeared" results typical of WPE.

Limitations and Future Work

  • Single Speaker Assumption: The current model assumes only one person is talking. In "cocktail party" scenarios with overlapping speakers, the model would need to be extended to perform separation and dereverberation simultaneously.
  • Stationary Noise: The "garbage source" approach works best for weak or stationary noise. Handling loud, non-stationary background noise remains a challenge.
  • Real-time Processing: The current study focuses on offline processing. Adapting the DNN to be causal for real-time applications is a suggested next step.

Summary updated

Abstract

In reverberant conditions with a single speaker, each far-field microphone records a reverberant version of the same speaker signal at a different location. In over-determined conditions, where there are multiple microphones but only one speaker, each recorded mixture signal can be leveraged as a constraint to narrow down the solutions to target anechoic speech and thereby reduce reverberation. Equipped with this insight, we propose USDnet, a novel deep neural network (DNN) approach for unsupervised speech dereverberation (USD). At each training step, we first feed an input mixture to USDnet to produce an estimate for target speech, and then linearly filter the DNN estimate to approximate the multi-microphone mixture so that the constraint can be satisfied at each microphone, thereby regularizing the DNN estimate to approximate target anechoic speech. The linear filter can be estimated based on the mixture and DNN estimate via neural forward filtering algorithms such as forward convolutive prediction. We show that this novel methodology can promote unsupervised dereverberation of single-source reverberant speech.
BibTeX
@article{wang2024usdnet,
  author = {Zhong-Qiu Wang},
  title = {{USDnet: Unsupervised Speech Dereverberation via Neural Forward Filtering}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2024},
  url = {https://openreview.net/forum?id=wes6sLbcj5},
}
Figure