TL;DR
This paper introduces ADEnet, a novel neural network for extracting a target speaker's voice from a single audio channel using only their speech activity (when they are speaking) as a clue. This approach bypasses the need for pre-recorded enrollment utterances, achieving competitive performance and significantly improving Automatic Speech Recognition (ASR) in challenging meeting scenarios with overlapping speakers.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems struggle when multiple people speak simultaneously, a common occurrence in meetings or conversations. Traditional speech separation methods often require knowing the exact number of speakers and can suffer from arbitrary speaker mapping. Target speech extraction aims to isolate a specific speaker's voice using auxiliary clues, but existing methods like SpeakerBeam rely on pre-recorded "enrollment" utterances of the target speaker. This is a significant limitation, as such recordings are not always available, and mismatches between enrollment and actual speech can degrade performance. This paper addresses the need for a more practical and flexible target speech extraction method that doesn't require enrollment data, focusing instead on readily available speaker activity information.
Methods
The core contribution is the Speaker Activity Driven Neural Speech Extraction (ADEnet). Unlike SpeakerBeam, which uses an enrollment utterance to generate a speaker embedding, ADEnet leverages a binary signal indicating when the target speaker is active (speaking or not). The authors propose three configurations of ADEnet:
- ADEnet-auxiliary: Similar to SpeakerBeam, but uses the target speaker's active segments from the mixture as input to an auxiliary network to generate a speaker embedding. This configuration aims to avoid recording condition mismatches by using segments where the target speaker is the only active one.
- ADEnet-input: This simpler approach directly concatenates the speaker activity signal with the mixture signal at the input of the extraction network. It bypasses the explicit speaker embedding generation, relying on the network to learn speaker identification internally.
- ADEnet-mix: A hybrid approach that combines both ADEnet-auxiliary and ADEnet-input, using speaker activity both for an embedding and at the input.
A crucial aspect of their method is robustness training. Recognizing that speaker activity information obtained in real-world scenarios (e.g., from diarization) is rarely perfect, they introduce noise to the oracle speaker activity during training. This involves slightly shifting the boundaries of the "speaking" segments, making the trained ADEnet more resilient to estimation errors during testing.
Datasets
- Simulated two-speaker mixtures: Created using the LibriSpeech corpus, with varying signal-to-noise ratios (SNRs) and overlap ratios. Used for initial experiments to demonstrate the core capabilities of ADEnet.
- LibriCSS corpus: An 8-speaker meeting-like recording dataset. This was used to evaluate ADEnet in a more realistic, challenging scenario and demonstrate its utility as a pre-processor for ASR.
Key Insights and Ideas
- Speaker activity is a powerful clue: The paper convincingly shows that knowing when a speaker is active is sufficient for effective target speech extraction, even without knowing who they are beforehand or having their voice sample.
- No enrollment needed, big practical win: Eliminating the need for enrollment utterances significantly broadens the applicability of target speech extraction in real-world scenarios where such data is scarce or impossible to obtain.
- Robustness training is key: The method's ability to handle imperfect speaker activity information (due to noise injection during training) is critical for its practical success, as real-world diarization systems are not perfect.
- Meeting scenarios benefit greatly: ADEnet, when combined with diarization, offers a practical and effective way to improve ASR performance in noisy, multi-speaker meeting environments, especially in high-overlap conditions.
Key Results
- Simulated Mixtures: ADEnet-auxiliary achieved a Signal-to-Distortion Ratio (SDR) of 10.2 dB with oracle speaker activity, outperforming SpeakerBeam (9.4 dB SDR) without requiring enrollment.
- Robustness: With noisy activity training, ADEnet-mix achieved an average SDR of 8.8 dB, demonstrating significant robustness to imperfect activity signals.
- Meeting Scenarios (LibriCSS): When used as a pre-processor for ASR, ADEnet consistently improved the concatenated minimum-permutation word error rate (cpWER) compared to using diarization alone. The relative Word Error Rate (WER) improvement ranged from 6.5% to 25%, particularly in high overlapping regions. This improvement is notable given ADEnet works with single-channel audio, whereas some prior multi-channel methods achieved similar or worse WERs.
Limitations and Future Work
- Limitations: The paper acknowledges that while ADEnet is robust to noisy speaker activity, its performance degrades when precise speaker activity is unavailable or when dealing with significant overlap. The current experiments focus on single-channel extraction.
- Future Work: The authors plan to explore more advanced network architectures, investigate multi-microphone extensions of ADEnet, and compare it with visual clue-based speech extraction approaches. They also suggest integrating ADEnet with existing multi-microphone systems like GSS.
Summary updated