Multi-Channel Block-Online Source Extraction Based on Utterance Adaptation

Juan M. Martín-Doñas, Jens Heitkaemper, Reinhold Haeb-Umbach, Angel M. Gomez, Antonio M. PeinadoInterspeech 2019

TL;DR

This paper introduces novel multi-channel speech extraction systems that leverage an adaptation utterance to help a neural network focus on a specific target speaker, even in noisy, reverberant environments with overlapping speakers. By incorporating both spectral and spatial information from the adaptation utterance, these systems achieve state-of-the-art performance in reducing distortion and improving recognition accuracy, particularly in challenging same-gender speaker scenarios.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems, especially in challenging environments like conference rooms, struggle with overlapping speech from multiple speakers. While deep neural networks have significantly improved ASR accuracy, their performance degrades when multiple speakers are active simultaneously. Existing methods for separating all speakers (like Deep Clustering or Deep Attractor Networks) are not ideal when the goal is to isolate and transcribe only one target speaker. Previous approaches to target speaker extraction often relied on predefined adaptation utterances or oracle information about speaker direction. This paper addresses the need for a more flexible and robust method that can adapt to a target speaker using any adaptation utterance and effectively separate them from interference, even when the interfering speakers have similar acoustic characteristics.

Methods

The core of the proposed systems builds upon the "Speaker Beam" (SB) approach, which uses an adaptation utterance (AU) to guide a neural network mask estimator. The authors introduce two novel variants that enhance this by incorporating spatial information:

  • PreBF (Spatial Pre-processing): This approach applies a Minimum Variance Distortionless Response (MVDR) beamformer as a spatial pre-processor to both the noisy speech signal and the AU before they are fed into the SB mask estimator and its auxiliary network. This pre-processing helps to suppress interfering speakers, making it easier for the subsequent neural network to estimate accurate masks for the target speaker.
  • Spatial + Spectral Features: This method directly feeds spatial and spectral features extracted from the AU into the auxiliary network and the SB mask estimator. Specifically, it uses Interchannel Phase Differences (IPD) as spatial features and the normalized logarithmic spectrum of the AU as spectral features. For the input to the SB mask estimator, it concatenates spectral and spatial features derived from the multi-channel noisy speech. This approach aims to provide richer speaker embeddings for better discrimination.

Both systems are designed for block-online processing, meaning they recursively update signal statistics in blocks of frames for low-latency operation. They also propose an improved initialization strategy for the spatial covariance matrices (SCMs) using diffuse noise and AU information, which helps in faster and more accurate beamformer convergence.

Datasets

The experiments were conducted on a simulated database described in [27]. This database consists of:

  • Training: 30,000 utterances.
  • Development: 500 utterances.
  • Evaluation: 1,500 utterances.

Each example was created by mixing two utterances from the Wall Street Journal (WSJ) database, convolved with six-channel Room Impulse Responses (RIRs) simulated using the Image Method. The signals were downsampled to 8 kHz. This setup creates an open-speaker-set database, meaning speakers in the evaluation set are not present in the training set, making it a more realistic test of generalization.

Key Insights and Ideas

  • Spatial information is crucial for same-gender speaker separation: The paper highlights that while spectral information alone can be ambiguous, especially between speakers of the same gender, incorporating spatial cues significantly improves the ability to distinguish and extract the target speaker.
  • Pre-processing can simplify the mask estimation task: The PreBF approach demonstrates that even a moderately effective spatial pre-filter can significantly aid a neural network mask estimator, especially in challenging scenarios like same-gender overlap.
  • Block-online processing is feasible and effective: The recursive updating of signal statistics allows for low-latency operation, making these systems suitable for real-time applications without a substantial performance penalty compared to offline methods.
  • Initialization matters: A well-chosen initialization for the spatial covariance matrices (using diffuse noise and AU information) leads to better performance and faster convergence, bridging the gap between proposed online methods and offline beamformers.

Key Results

  • Improved WER and SDR: Both proposed systems (PreBF and Spatial+Spectral Features) outperform state-of-the-art approaches like DANs and SpeakerBeam (SB) in terms of Word Error Rate (WER) and Signal-to-Distortion Ratio (SDR).
  • Superior performance in same-gender scenarios: The PreBF approach, in particular, shows a significant reduction in WER degradation for same-gender speaker overlap, achieving results comparable to the different-gender case. For WER, the difference between different and same-gender utterances was reduced from 11.69% to 4.46% with PreBF.
  • Competitive intelligibility: Both systems achieve better speech intelligibility gains (STOI) compared to existing methods.
  • Online vs. Offline: While online versions generally have a higher WER due to block-online updates, they still achieve competitive results for online recognition, with the spatial features approach being preferred for ASR.

Limitations and Future Work

  • Limitations:
    • The performance of online systems is slightly degraded compared to offline methods due to the block-online updating of SCM statistics and the use of a single LSTM layer.
  • While spatial features improve accuracy, the network still faces challenges in learning both spectral and spatial characteristics for separation in same-gender scenarios.
  • The assumption of a steady target speaker position is made, although the paper notes that a short reinitialization can adapt to minor position changes.
  • Future Work:
    • The authors suggest exploring combinations of the proposed systems to achieve further improvements.
  • Investigating more advanced techniques for handling speaker position changes would be beneficial.

Summary updated

BibTeX
@inproceedings{martindonas2019multi,
  author = {Juan M. Martín-Doñas and Jens Heitkaemper and Reinhold Haeb-Umbach and Angel M. Gomez and Antonio M. Peinado},
  title = {{Multi-Channel Block-Online Source Extraction Based on Utterance Adaptation}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure