Abstract
In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embedding as input, and produces a mask. Our system significantly reduces the speech recognition WER on multi-speaker signals, with minimal WER degradation on single-speaker signals.
Introduction: Solving the Cocktail Party Problem
Automated Speech Recognition (ASR) systems have achieved human-level performance in quiet settings but still struggle with the "cocktail party problem"—isolating a single voice in a room with multiple overlapping speakers. Traditional source separation techniques often face hurdles such as not knowing the number of speakers or the "permutation problem," where the model confuses which output track belongs to which speaker.
VoiceFilter bypasses these issues by framing the task as voice filtering. Instead of general separation, it uses a short reference recording of a target speaker to "filter" their voice out of a noisy, multi-speaker stream.
Methods: Speaker-Conditioned Masking
The VoiceFilter architecture consists of two primary components: a pre-trained speaker encoder and a trainable masking network.
- Speaker Encoder: A pre-trained 3-layer LSTM network that takes a reference audio clip and produces a d-vector—a 256-dimensional embedding capturing unique vocal characteristics.
- VoiceFilter Network: This masking engine takes the magnitude spectrogram of the noisy audio and the target speaker's d-vector. It uses 8 dilated convolutional layers to capture wide acoustic context, followed by LSTM and fully connected layers. The final output is a soft mask applied to the noisy spectrogram to suppress everything except the target speaker.
Datasets and Training
The system was evaluated using two primary datasets:
- LibriSpeech: Used for large-scale training. The authors created "triplets" by mixing a clean utterance with random interference and providing a separate reference clip.
- VCTK: Used to test generalization to new speakers and recording conditions.
- Internal Google Data: The speaker encoder was trained on 34 million anonymized voice queries to ensure robust d-vectors.
Key Results
The performance gains were significant, particularly when using a bi-directional LSTM in the VoiceFilter network.
On the LibriSpeech dataset, VoiceFilter achieved a 58.1% relative reduction in Word Error Rate (WER) for overlapped speech, bringing it down from 55.9% to 23.4%. Crucially, the system had a minimal negative impact on clean audio, with WER only increasing from 10.9% to 11.1%. In terms of signal quality, VoiceFilter outperformed the "Permutation Invariant Training" (PIT) baseline, achieving a higher Source to Distortion Ratio (SDR) of 17.9 dB compared to 17.2 dB for the baseline and 10.1 dB for unprocessed audio.
Conclusion
VoiceFilter provides a robust solution for targeted voice separation by leveraging speaker embeddings. By conditioning the masking process on a specific biometric hint, the model avoids the complexities of general source separation. Future work will focus on incorporating more diverse ambient noises and exploring joint optimization of the encoder and filter for further ASR improvements.
Summary updated