All-neural beamformer for continuous speech separation

Zhuohuang Zhang, Takuya Yoshioka, Naoyuki Kanda, Zhuo Chen, Xiaofei Wang, Dongmei Wang, Sefik Emre Eskimez

Abstract

Continuous speech separation (CSS) aims to separate overlapping voices from a continuous influx of conversational audio containing an unknown number of utterances spoken by an unknown number of speakers. A common application scenario is transcribing a meeting conversation recorded by a microphone array. Prior studies explored various deep learning models for time-frequency mask estimation, followed by a minimum variance distortionless response (MVDR) filter to improve the automatic speech recognition (ASR) accuracy. The performance of these methods is fundamentally upper-bounded by MVDR's spatial selectivity. Recently, the all deep learning MVDR (ADL-MVDR) model was proposed for neural beamforming and demonstrated superior performance in a target speech extraction task using pre-segmented input. In this paper, we further adapt ADL-MVDR to the CSS task with several enhancements to enable end-to-end neural beamforming. The proposed system achieves significant word error rate reduction over a baseline spectral masking system on the LibriCSS dataset. Moreover, the proposed neural beamformer is shown to be comparable to a state-of-the-art MVDR-based system in real meeting transcription tasks, including AMI, while showing potentials to further simplify the runtime implementation and reduce the system latency with frame-wise processing.

Introduction

In real-world conversational environments, such as office meetings, overlapping speech is a frequent occurrence that significantly degrades the performance of Automatic Speech Recognition (ASR) systems. This "cocktail party" problem is typically addressed using Speech Separation as a front-end. While purely deep-learning-based separation can isolate voices, it often introduces non-linear distortions that negatively impact ASR backends.

To mitigate this, the industry has traditionally relied on the Minimum Variance Distortionless Response (MVDR) filter, which utilizes spatial information to isolate speakers while preserving speech naturalness. However, traditional MVDR implementations face two primary challenges: numerical instability during backpropagation due to matrix inversions, and a lack of temporal granularity, as they often average spatial statistics over long segments (chunks). This paper proposes a stable, fully neural beamformer based on the ADL-MVDR framework, adapted for continuous, unsegmented audio.

Methodology

The proposed system replaces traditional mathematical matrix operations with a neural architecture capable of frame-wise processing. The pipeline consists of several integrated components.

System Architecture

The process begins with a T-F Mask Estimator, utilizing a Conformer-based network to predict Time-Frequency masks for potential speakers and background noise from multi-channel audio. These masks are then fed into the ADL-MVDR module. Unlike traditional beamformers, ADL-MVDR uses Gated Recurrent Units (GRUs) to estimate the steering vector and the inverse noise covariance matrix on a frame-by-frame basis.

Overall architecture of the proposed neural beamforming system for CSS.

Key Enhancements for Stability and Performance

To ensure the neural beamformer remains mathematically well-behaved and effective in continuous streams, several enhancements were introduced:

  • Steering Vector Normalization: Stabilizes directional estimates.
  • Positive Semi-Definite (PSD) Constraint: The inverse noise covariance matrix is forced to be the product of a triangular matrix and its transpose ($UU^H$), preventing numerical explosions.
  • Frame-wise VAD: A dedicated GRU-based Voice Activity Detector suppresses noise during silent intervals, preventing the beamformer from amplifying artifacts when no speaker is active.
  • Residual Connection: A portion of the original masked signal is added back to the output to maintain speech naturalness.

Experimental Setup

The model was trained on 219 hours of simulated multi-channel mixtures from the WSJ1 corpus, augmented with MUSAN noise. Evaluation was conducted on the LibriCSS benchmark (10 hours of multi-talker recordings with 0-40% overlap) and real-world datasets including the AMI meeting corpus and a Microsoft internal meeting dataset (MS).

Results and Discussion

The proposed neural beamformer demonstrated superior performance across various benchmarks. On the LibriCSS dataset, the system achieved an average Word Error Rate (WER) of 10.1%, representing a 9% relative reduction over the baseline spectral masking system.

Word Error Rate (WER) results on LibriCSS across different overlap ratios.

As shown in the results table, the gains are particularly significant in high-overlap scenarios. In the 30% overlap condition, the neural beamformer provided a 15% relative WER improvement. Ablation studies highlight the importance of the VAD and normalization components in achieving these results.

In real-world meeting scenarios, the system remained competitive. On the Microsoft internal dataset, the ADL-MVDR system (16.3% WER) outperformed both a standard beamformer (17.8%) and a state-of-the-art conventional MVDR system (16.6%). The frame-wise nature of the GRU-based estimation allows the system to adapt more rapidly to changing acoustic environments compared to traditional chunk-based MVDR.

Conclusion

This work presents a robust, all-neural beamformer for continuous speech separation. By integrating mathematical constraints into a deep learning framework, the system achieves state-of-the-art performance while avoiding the numerical instability of traditional filters. Future work will focus on scaling the system to handle more than two simultaneous speakers and further optimizing runtime complexity for real-time deployment.

Summary updated

Abstract

Continuous speech separation (CSS) aims to separate overlapping voices from a continuous influx of conversational audio containing an unknown number of utterances spoken by an unknown number of speakers. A common application scenario is transcribing a meeting conversation recorded by a microphone array. Prior studies explored various deep learning models for time-frequency mask estimation, followed by a minimum variance distortionless response (MVDR) filter to improve the automatic speech recognition (ASR) accuracy. The performance of these methods is fundamentally upper-bounded by MVDR's spatial selectivity. Recently, the all deep learning MVDR (ADL-MVDR) model was proposed for neural beamforming and demonstrated superior performance in a target speech extraction task using pre-segmented input. In this paper, we further adapt ADL-MVDR to the CSS task with several enhancements to enable end-to-end neural beamforming. The proposed system achieves significant word error rate reduction over a baseline spectral masking system on the LibriCSS dataset. Moreover, the proposed neural beamformer is shown to be comparable to a state-of-the-art MVDR-based system in real meeting transcription tasks, including AMI, while showing potentials to further simplify the runtime implementation and reduce the system latency with frame-wise processing.
BibTeX
@inproceedings{zhang2022all,
  author = {Zhuohuang Zhang and Takuya Yoshioka and Naoyuki Kanda and Zhuo Chen and Xiaofei Wang and Dongmei Wang and Sefik Emre Eskimez},
  title = {{All-neural beamformer for continuous speech separation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=gQ8c-Em9tdp},
}
Figure