VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking

Quan Wang, Hannah Muckenhirn, Kevin W. Wilson, Prashant Sridhar, Zelin Wu, John R. Hershey, Rif A. Saurous, Ron J. Weiss, Ye Jia, Ignacio Lopez-Moreno

Abstract

In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embedding as input, and produces a mask. Our system significantly reduces the speech recognition WER on multi-speaker signals, with minimal WER degradation on single-speaker signals.

Introduction: Solving the Cocktail Party Problem

Automated Speech Recognition (ASR) systems have achieved human-level performance in quiet settings but still struggle with the "cocktail party problem"—isolating a single voice in a room with multiple overlapping speakers. Traditional source separation techniques often face hurdles such as not knowing the number of speakers or the "permutation problem," where the model confuses which output track belongs to which speaker.

VoiceFilter bypasses these issues by framing the task as voice filtering. Instead of general separation, it uses a short reference recording of a target speaker to "filter" their voice out of a noisy, multi-speaker stream.

Methods: Speaker-Conditioned Masking

The VoiceFilter architecture consists of two primary components: a pre-trained speaker encoder and a trainable masking network.

VoiceFilter system architecture showing the speaker encoder and masking network.
  1. Speaker Encoder: A pre-trained 3-layer LSTM network that takes a reference audio clip and produces a d-vector—a 256-dimensional embedding capturing unique vocal characteristics.
  2. VoiceFilter Network: This masking engine takes the magnitude spectrogram of the noisy audio and the target speaker's d-vector. It uses 8 dilated convolutional layers to capture wide acoustic context, followed by LSTM and fully connected layers. The final output is a soft mask applied to the noisy spectrogram to suppress everything except the target speaker.

Datasets and Training

The system was evaluated using two primary datasets:

  • LibriSpeech: Used for large-scale training. The authors created "triplets" by mixing a clean utterance with random interference and providing a separate reference clip.
  • VCTK: Used to test generalization to new speakers and recording conditions.
  • Internal Google Data: The speaker encoder was trained on 34 million anonymized voice queries to ensure robust d-vectors.

Key Results

The performance gains were significant, particularly when using a bi-directional LSTM in the VoiceFilter network.

Source to Distortion Ratio (SDR) results comparing VoiceFilter variants with a baseline.

On the LibriSpeech dataset, VoiceFilter achieved a 58.1% relative reduction in Word Error Rate (WER) for overlapped speech, bringing it down from 55.9% to 23.4%. Crucially, the system had a minimal negative impact on clean audio, with WER only increasing from 10.9% to 11.1%. In terms of signal quality, VoiceFilter outperformed the "Permutation Invariant Training" (PIT) baseline, achieving a higher Source to Distortion Ratio (SDR) of 17.9 dB compared to 17.2 dB for the baseline and 10.1 dB for unprocessed audio.

Conclusion

VoiceFilter provides a robust solution for targeted voice separation by leveraging speaker embeddings. By conditioning the masking process on a specific biometric hint, the model avoids the complexities of general source separation. Future work will focus on incorporating more diverse ambient noises and exploring joint optimization of the encoder and filter for further ASR improvements.

Summary updated

Abstract

In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embedding as input, and produces a mask. Our system significantly reduces the speech recognition WER on multi-speaker signals, with minimal WER degradation on single-speaker signals.
BibTeX
@inproceedings{wang2019voicefilter,
  author = {Quan Wang and Hannah Muckenhirn and Kevin W. Wilson and Prashant Sridhar and Zelin Wu and John R. Hershey and Rif A. Saurous and Ron J. Weiss and Ye Jia and Ignacio Lopez-Moreno},
  title = {{VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=rGcT6X7VEy2},
}
Figure