Multichannel Loss Function for Supervised Speech Source Separation by Mask-Based Beamforming

Yoshiki Masuyama, Masahito Togami, Tatsuya KomatsuInterspeech 2019

TL;DR

This paper introduces novel multichannel loss functions for training deep neural networks (DNNs) to improve speech source separation using mask-based beamforming. By directly optimizing for the accuracy of spatial covariance matrices, these new loss functions lead to better separation performance compared to traditional monaural loss functions, especially in challenging acoustic environments.

What Problem Does This Paper Address?

Speech source separation, the task of isolating individual speech signals from a mixture, is crucial for applications like hearing aids and automatic speech recognition (ASR). While multichannel microphone arrays offer significant advantages over single-microphone systems by leveraging spatial information, effectively utilizing this information in supervised separation remains a challenge. Existing methods often train DNNs for mask-based beamforming using monaural loss functions (e.g., Phase Sensitive Approximation - PSA). However, these monaural losses, which focus on individual time-frequency bins, don't directly optimize for the accuracy of spatial covariance matrices. These matrices are fundamental for constructing beamformers, and their inaccurate estimation can degrade separation performance. This paper aims to bridge this gap by proposing multichannel loss functions that directly evaluate and optimize the estimated spatial covariance matrices.

Methods

The core contribution of this paper lies in the development and application of two novel multichannel loss functions for mask-based beamforming.

  • Mask-Based Beamforming: This approach uses a DNN to estimate time-frequency (TF) masks. These masks are then used to estimate spatial covariance matrices, which in turn are used to construct various beamformers (e.g., MVDR, GEV, MWF). A key advantage is that this method is independent of the specific microphone configuration used during testing, as long as the DNN is trained appropriately.
  • Multichannel Loss Function 1 (Prop. 1): This method adapts a previously proposed multichannel loss function for time-varying Multichannel Wiener Filter (MWF) [28] to the time-invariant mask-based beamforming context. It uses the multichannel Itakura-Saito divergence (MISD) to evaluate the estimated spatial covariance matrices. The DNN in this approach is trained to estimate both time-varying activation and time-invariant spatial covariance matrices.
  • Multichannel Loss Function 2 (Prop. 2): Recognizing that the time-varying activation estimation is redundant for time-invariant beamforming and computationally expensive, the authors propose a more efficient multichannel loss function. This second approach also uses MISD but avoids the explicit estimation of time-varying activation, leading to reduced computational cost and a simpler DNN architecture.

Both proposed methods leverage Permutation Invariant Training (PIT) to handle the utterance-level permutation problem inherent in multi-talker separation.

Datasets

The experiments were conducted using:

  • Multichannel Impulse Response Database (MIRD) [33]: Used for generating multichannel signals by convolving clean speech with measured impulse responses.
  • TIMIT corpus [34]: Provided the clean speech signals.

The training and testing were performed under three different conditions, varying microphone arrangements (closed-set and open-set relative to training) and reverberation times (RT60 of 160 ms and 360 ms).

Key Insights and Ideas

  • Directly optimizing spatial covariance matrices is crucial: The paper strongly suggests that monaural loss functions, while simple, are suboptimal because they don't directly target the accuracy of spatial covariance matrices, which are the foundation of beamforming.
  • Multichannel losses improve robustness: The proposed multichannel loss functions lead to improved separation performance across different microphone configurations and reverberation conditions, demonstrating their generalization capabilities.
  • Computational efficiency matters: While the first proposed method (Prop. 1) shows excellent results, the second method (Prop. 2) offers a compelling trade-off between performance and computational cost by removing redundant estimations.
  • Mask-based beamforming is versatile: The effectiveness of the proposed multichannel losses is demonstrated across multiple beamformer types (MVDR, GEV, MWF), highlighting the flexibility of the mask-based approach.

Key Results

The paper reports significant improvements in speech separation metrics (SDR, SIR, CD) when using the proposed multichannel loss functions compared to the monaural PSA loss.

  • Condition 1 (Closed mic-arrangement, RT60=160ms): Prop. 1 achieved the highest SDR (8.54 dB) and SIR (9.42 dB) for the MVDR beamformer, outperforming PSA [10] and even approaching the performance of the time-varying [28] method. Prop. 2 also showed strong results, slightly below Prop. 1.

  • Condition 2 (Open mic-arrangement, RT60=160ms): Similar trends were observed, with Prop. 1 and Prop. 2 consistently outperforming PSA.

  • Condition 3 (Open mic-arrangement, RT60=360ms): In more challenging reverberant conditions, the proposed methods still demonstrated superior performance over PSA, although the gap narrowed compared to less reverberant scenarios.
  • Computational Cost: Prop. 1 was found to be 3.1 times slower to train than Prop. 2 due to the additional computation for time-varying MWF.

Limitations and Future Work

  • Computational Cost of Prop. 1: While Prop. 1 achieves excellent results, its training is significantly slower than Prop. 2.

  • Oracle PSM Benchmark: The Oracle PSM (Phase Sensitive Mask) serves as a strong upper bound, and while the proposed methods approach its performance, there's still room for improvement.

  • Future Work: The authors suggest that the multichannel loss functions can be applied to other mask-based beamforming techniques. Further exploration into optimizing the trade-off between performance and computational complexity for different applications is also implied.

Summary updated

BibTeX
@inproceedings{masuyama2019multichannel,
  author = {Yoshiki Masuyama and Masahito Togami and Tatsuya Komatsu},
  title = {{Multichannel Loss Function for Supervised Speech Source Separation by Mask-Based Beamforming}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure