Abstract
In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embedding as input, and produces a mask. Our system significantly reduces the speech recognition WER on multi-speaker signals, with minimal WER degradation on single-speaker signals.
Introduction
Automatic Speech Recognition (ASR) has reached high levels of accuracy in quiet environments but continues to struggle with the "Cocktail Party Problem"—the challenge of isolating a single voice amidst competing speakers. Traditional speech separation techniques often attempt to partition audio into $N$ separate tracks. However, these methods face significant hurdles, including an unknown number of speakers, the "permutation problem" where speaker tracks are swapped mid-recording, and the difficulty of selecting the correct target track for subsequent ASR processing.
VoiceFilter addresses these challenges by reframing the task as a targeted filtering problem. Instead of general separation, the system uses a reference recording of a target speaker to "filter" the audio, isolating the desired voice from all other interference.
Methods
The VoiceFilter architecture consists of two primary components, which are trained independently to optimize their respective tasks.
Speaker Encoder
A pre-trained LSTM network serves as the speaker encoder. It processes a short reference "enrollment" audio sample from the target speaker to generate a d-vector. This 256-dimensional embedding acts as a unique numerical "voiceprint" that captures the essential acoustic characteristics of the speaker's identity.
VoiceFilter Network
The core of the system is the VoiceFilter network, which takes the noisy magnitude spectrogram of the mixed audio and the target speaker's d-vector as inputs.
- Feature Extraction: Eight convolutional layers, including dilated convolutions to capture broader acoustic context, extract features from the input spectrogram.
- Conditioning: The d-vector is concatenated with the CNN outputs, providing the network with the target identity it needs to isolate.
- Mask Prediction: Subsequent LSTM and fully connected layers process the conditioned features to predict a soft mask—a matrix of values between 0 and 1.
- Reconstruction: The predicted mask is element-wise multiplied by the original noisy spectrogram. The resulting "masked spectrogram" is then combined with the original phase information and converted back into the time domain via Inverse Short-Time Fourier Transform (ISTFT) to produce the enhanced audio.
Experiments and Results
The system was evaluated using datasets like LibriSpeech and VCTK, with artificial mixtures created to simulate multi-speaker interference. A key finding was that separate training of the encoder and filter was more effective than joint training, as speaker recognition benefits from massive speaker diversity while separation requires paired clean/noisy data.
Performance Metrics
The primary evaluation metric was the Word Error Rate (WER) of an ASR system processing the filtered audio. On the LibriSpeech dataset, the system demonstrated a dramatic reduction in WER for noisy, multi-speaker environments.
As shown in the results, the bi-LSTM version of VoiceFilter reduced the WER on noisy audio from 55.9% to 23.4%, a relative improvement of over 58%. Importantly, the WER on clean (single-speaker) audio only increased marginally from 10.9% to 11.1%, indicating that the system is "safe" to use even when no interference is present.
Conclusion
VoiceFilter represents a significant step forward in solving the cocktail party problem for ASR. By leveraging speaker-discriminative embeddings, the system avoids the complexities of blind source separation and provides a targeted, robust method for voice enhancement. Future work may involve joint optimization with ASR systems and extending the model to handle ambient noise suppression simultaneously.
Summary updated