Multichannel Speech Enhancement without Beamforming

Ashutosh Pandey, Buye Xu, Anurag Kumar, Jacob Donley, Paul Calamia, DeLiang Wang

Abstract

Deep neural networks are often coupled with traditional spatial filters, such as MVDR beamformers, for effectively exploiting spatial information. Even though single-stage end-to-end supervised models can obtain impressive enhancement, combining them with a traditional beamformer and a DNN-based post-filter in a multistage processing provides additional improvements. In this work, we propose a two-stage strategy for multi-channel speech enhancement that does not require a traditional beamformer for additional performance. First, we propose a novel attentive dense convolutional network (ADCN) for estimating real and imaginary parts of complex spectrogram. ADCN obtains state-of-the-art results among single-stage models. Next, we use ADCN with a recently proposed triple-path attentive recurrent network (TPARN) for estimating waveform samples. The proposed strategy uses two insights; first, using different approaches in two stages; and second, using a stronger model in the first stage. We illustrate the efficacy of our strategy by evaluating multiple models in a two-stage approach with and without a traditional beamformer.

Introduction

In multichannel speech enhancement, the primary objective is to extract clean speech from recordings captured by multiple microphones in noisy, reverberant environments. Historically, the most effective systems have been hybrid: they use Deep Neural Networks (DNNs) to estimate signal statistics, which are then processed by a linear spatial filter, such as a Minimum Variance Distortionless Response (MVDR) beamformer.

While MVDR beamformers are mathematically "distortionless," they require complex matrix inversions that can be numerically unstable during training. This research challenges the necessity of these traditional filters, asking whether a purely neural architecture can achieve similar or superior results by implicitly learning spatial cues.

The Attentive Dense Convolutional Network (ADCN)

The authors introduce the Attentive Dense Convolutional Network (ADCN) as a powerful single-stage model. Built on a UNet-style architecture, ADCN utilizes Dense Blocks to preserve feature flow across layers and Attention Blocks to capture long-range dependencies across time and frequency. Unlike models that only predict magnitude, ADCN performs "complex spectral mapping," predicting both the real and imaginary parts of the clean speech spectrogram.

The architecture of the proposed Attentive Dense Convolutional Network (ADCN), featuring an encoder-decoder structure with dense and attention blocks.

A Two-Stage "Hybrid Mapping" Strategy

The core innovation of this work is a two-stage strategy that replaces the traditional beamformer with a second neural stage. The strategy relies on two key principles:

  1. Domain Diversity: Stage 1 performs complex spectral mapping (frequency domain), while Stage 2 performs waveform mapping (time domain). Because these approaches have different mathematical biases, the second stage can effectively "fix" the non-linear distortions introduced by the first.
  2. Strong-to-Weak Ordering: Placing the stronger model (ADCN) in the first stage yields significantly better results than the reverse.
Comparison of the two-stage approach (a) with a traditional TI-MVDR beamformer and (b) without a beamformer, where the second stage directly processes the output of the first.

Experimental Results

The proposed models were evaluated on the WSJCAM0 + REVERB Challenge and the DNS 2020 datasets.

  • Single-Stage Performance: ADCN established a new state-of-the-art for single-stage models, outperforming existing architectures like DCRN and TPARN in both speech quality (PESQ) and intelligibility (STOI).
  • Redundancy of Beamformers: Crucially, the results showed that in a two-stage system, adding an MVDR beamformer between stages provided no significant benefit. The neural stages learned to exploit spatial information implicitly.
  • Quantitative Gains: On the difficult DNS dataset, the two-stage ADCN + TPARN approach reached an SI-SDR of 10.0 dB, a massive improvement over the unprocessed -7.6 dB and the single-stage ADCN's 7.8 dB.

Conclusion

This work demonstrates that traditional beamformers are not a requirement for high-quality multichannel speech enhancement. By stacking diverse neural mapping strategies—specifically spectral mapping followed by waveform mapping—researchers can achieve state-of-the-art, distortion-free results through purely learned spatial filtering. Future work will focus on reducing the computational complexity of these models for real-time applications on edge devices.

Summary updated

Abstract

Deep neural networks are often coupled with traditional spatial filters, such as MVDR beamformers for effectively exploiting spatial information. Even though single-stage end-to-end supervised models can obtain impressive enhancement, combining them with a traditional beamformer and a DNN-based post-filter in a multistage processing provides additional improvements. In this work, we propose a two-stage strategy for multi-channel speech enhancement that does not require a traditional beamformer for additional performance. First, we propose a novel attentive dense convolutional network (ADCN) for estimating real and imaginary parts of complex spectrogram. ADCN obtains state-of-the-art results among single-stage models. Next, we use ADCN with a recently proposed triple-path attentive recurrent network (TPARN) for estimating waveform samples. The proposed strategy uses two insights; first, using different approaches in two stages; and second, using a stronger model in the first stage. We illustrate the efficacy of our strategy by evaluating multiple models in a two-stage approach with and without a traditional beamformer.
BibTeX
@inproceedings{pandey2022multichannel,
  author = {Ashutosh Pandey and Buye Xu and Anurag Kumar and Jacob Donley and Paul Calamia and DeLiang Wang},
  title = {{Multichannel Speech Enhancement without Beamforming}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=NvxgUr8AavM},
}
Figure