Speaker Activity Driven Neural Speech Extraction

Marc Delcroix, Katerina Zmolíková, Tsubasa Ochiai, Keisuke Kinoshita, Tomohiro Nakatani

TL;DR

This paper introduces ADEnet, a novel neural network for extracting a target speaker's voice from a single audio channel using only their speech activity (when they are speaking) as a clue. This approach bypasses the need for pre-recorded enrollment utterances, achieving competitive performance and significantly improving Automatic Speech Recognition (ASR) in challenging meeting scenarios with overlapping speakers.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems struggle when multiple people speak simultaneously, a common occurrence in meetings or conversations. Traditional speech separation methods often require knowing the exact number of speakers and can suffer from arbitrary speaker mapping. Target speech extraction aims to isolate a specific speaker's voice using auxiliary clues, but existing methods like SpeakerBeam rely on pre-recorded "enrollment" utterances of the target speaker. This is a significant limitation, as such recordings are not always available, and mismatches between enrollment and actual speech can degrade performance. This paper addresses the need for a more practical and flexible target speech extraction method that doesn't require enrollment data, focusing instead on readily available speaker activity information.

Methods

The core contribution is the Speaker Activity Driven Neural Speech Extraction (ADEnet). Unlike SpeakerBeam, which uses an enrollment utterance to generate a speaker embedding, ADEnet leverages a binary signal indicating when the target speaker is active (speaking or not). The authors propose three configurations of ADEnet:

  • ADEnet-auxiliary: Similar to SpeakerBeam, but uses the target speaker's active segments from the mixture as input to an auxiliary network to generate a speaker embedding. This configuration aims to avoid recording condition mismatches by using segments where the target speaker is the only active one.
  • ADEnet-input: This simpler approach directly concatenates the speaker activity signal with the mixture signal at the input of the extraction network. It bypasses the explicit speaker embedding generation, relying on the network to learn speaker identification internally.
  • ADEnet-mix: A hybrid approach that combines both ADEnet-auxiliary and ADEnet-input, using speaker activity both for an embedding and at the input.

A crucial aspect of their method is robustness training. Recognizing that speaker activity information obtained in real-world scenarios (e.g., from diarization) is rarely perfect, they introduce noise to the oracle speaker activity during training. This involves slightly shifting the boundaries of the "speaking" segments, making the trained ADEnet more resilient to estimation errors during testing.

Datasets

  • Simulated two-speaker mixtures: Created using the LibriSpeech corpus, with varying signal-to-noise ratios (SNRs) and overlap ratios. Used for initial experiments to demonstrate the core capabilities of ADEnet.
  • LibriCSS corpus: An 8-speaker meeting-like recording dataset. This was used to evaluate ADEnet in a more realistic, challenging scenario and demonstrate its utility as a pre-processor for ASR.

Key Insights and Ideas

  • Speaker activity is a powerful clue: The paper convincingly shows that knowing when a speaker is active is sufficient for effective target speech extraction, even without knowing who they are beforehand or having their voice sample.
  • No enrollment needed, big practical win: Eliminating the need for enrollment utterances significantly broadens the applicability of target speech extraction in real-world scenarios where such data is scarce or impossible to obtain.
  • Robustness training is key: The method's ability to handle imperfect speaker activity information (due to noise injection during training) is critical for its practical success, as real-world diarization systems are not perfect.
  • Meeting scenarios benefit greatly: ADEnet, when combined with diarization, offers a practical and effective way to improve ASR performance in noisy, multi-speaker meeting environments, especially in high-overlap conditions.

Key Results

  • Simulated Mixtures: ADEnet-auxiliary achieved a Signal-to-Distortion Ratio (SDR) of 10.2 dB with oracle speaker activity, outperforming SpeakerBeam (9.4 dB SDR) without requiring enrollment.
  • Robustness: With noisy activity training, ADEnet-mix achieved an average SDR of 8.8 dB, demonstrating significant robustness to imperfect activity signals.
  • Meeting Scenarios (LibriCSS): When used as a pre-processor for ASR, ADEnet consistently improved the concatenated minimum-permutation word error rate (cpWER) compared to using diarization alone. The relative Word Error Rate (WER) improvement ranged from 6.5% to 25%, particularly in high overlapping regions. This improvement is notable given ADEnet works with single-channel audio, whereas some prior multi-channel methods achieved similar or worse WERs.

Limitations and Future Work

  • Limitations: The paper acknowledges that while ADEnet is robust to noisy speaker activity, its performance degrades when precise speaker activity is unavailable or when dealing with significant overlap. The current experiments focus on single-channel extraction.
  • Future Work: The authors plan to explore more advanced network architectures, investigate multi-microphone extensions of ADEnet, and compare it with visual clue-based speech extraction approaches. They also suggest integrating ADEnet with existing multi-microphone systems like GSS.

Summary updated

Abstract

Target speech extraction, which extracts the speech of a target speaker in a mixture given auxiliary speaker clues, has recently received increased interest. Various clues have been investigated such as pre-recorded enrollment utterances, direction information, or video of the target speaker. In this paper, we explore the use of speaker activity information as an auxiliary clue for single-channel neural network-based speech extraction. We propose a speaker activity driven speech extraction neural network (ADEnet) and show that it can achieve performance levels competitive with enrollment-based approaches, without the need for pre-recordings. We further demonstrate the potential of the proposed approach for processing meeting-like recordings, where the speaker activity is obtained from a diarization system. We show that this simple yet practical approach can successfully extract speakers after diarization, which results in improved ASR performance, especially in high overlapping conditions, with a relative word error rate reduction of up to 25%.
BibTeX
@inproceedings{delcroix2021speaker,
  author = {Marc Delcroix and Katerina Zmolíková and Tsubasa Ochiai and Keisuke Kinoshita and Tomohiro Nakatani},
  title = {{Speaker Activity Driven Neural Speech Extraction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414998},
}
Figure