X-TaSNet: Robust and Accurate Time-Domain Speaker Extraction Network

Zining Zhang, Bingsheng He, Zhenjie Zhang

Abstract

Extracting the speech of a target speaker from mixed audios, based on a reference speech from the target speaker, is a challenging yet powerful technology in speech processing. Recent studies of speaker-independent speech separation, such as TasNet, have shown promising results by applying deep neural networks over the time-domain waveform. Such separation neural network does not directly generate reliable and accurate output when target speakers are specified, because of the necessary prior on the number of speakers and the lack of robustness when dealing with audios with absent speakers. In this paper, we break these limitations by introducing a new speaker-aware speech masking method, called X-TaSNet. Our proposal adopts new strategies, including a distortion-based loss and corresponding alternating training scheme, to better address the robustness issue. X-TaSNet significantly enhances the extracted speech quality, doubling SDRi and SI-SNRi of the output speech audio over state-of-the-art voice filtering approach. X-TaSNet also improves the reliability of the results by improving the accuracy of speaker identity in the output audio to 95.4%, such that it returns silent audios in most cases when the target speaker is absent. These results demonstrate X-TaSNet moves one solid step towards more practical applications of speaker extraction technology.

What Problem Does This Paper Address?

While speech separation (splitting a recording into $N$ different tracks) has seen massive gains recently, speaker extraction is a different beast. In extraction, you provide the model with a "clue"—a reference audio of the person you want to hear—and the model must find and isolate that specific voice from a mixture.

Existing solutions faced three major hurdles:

  1. The "Phase" Problem: Most previous models (like VoiceFilter) worked on spectrograms (frequency domain). This approach discards phase information, which creates a "ceiling" on how clear the output can be.
  2. Speaker Blindness: High-performing time-domain models like TasNet are "speaker-independent," meaning they can split voices but don't know which one belongs to whom.
  3. The Absence Dilemma: Real-world systems often fail when the target speaker isn't actually in the room. Instead of being silent, most models will mistakenly "extract" the person who sounds most like the target, leading to high error rates.

Methods

X-TaSNet builds upon the Conv-TasNet architecture but adapts it for targeted extraction. The system consists of four main parts:

  • Encoder/Decoder: Converts raw 1D waveforms into a high-dimensional internal representation and back again.
  • Speaker Encoder: A pre-trained module (using the GE2E algorithm) that creates a "voiceprint" embedding from the reference audio.
  • Extractor: A Temporal Convolutional Network (TCN) that takes both the noisy mixture and the voiceprint to create a "mask" that filters out unwanted sounds.
The X-TaSNet architecture diagram showing the flow from reference and noisy speech to the estimated target speech.

The researchers introduced three key training innovations:

  • Loss on Distortion (LoD): Instead of just calculating the error for the target speaker, the model is also forced to minimize the error for the "distortion" (the mixture of everyone else). This encourages the model to perform a clean separation rather than a messy "best-guess" extraction.
  • Alternating Training (AT): To improve data efficiency, the model is trained to treat every speaker in a mixture as a target in different iterations, ensuring it learns to distinguish between similar-sounding voices.
  • Speaker Presence Invariant Training (SPIT): The authors intentionally included training samples where the target speaker was absent. They introduced a "Decibel Loss" to penalize the model if it didn't output near-total silence in these cases.

Key Results

X-TaSNet outperformed the baseline (VoiceFilter) by a staggering margin:

  • Audio Quality: It achieved an SDRi (Signal-to-Distortion Ratio improvement) of 14.7 dB, nearly double the 7.4 dB of the previous state-of-the-art.
  • Accuracy: The Speaker Error Rate (SpkER)—the frequency with which the model extracts the wrong person—dropped from 9.5% to 4.6%.
Performance comparison of X-TaSNet variants against the VoiceFilter baseline across multiple metrics.
  • Absence Detection: In scenarios where the target speaker was missing, the specialized "X-TaSNet-PIT" model achieved a 72.4% Negative Energy Rate (NER), meaning it successfully stayed silent most of the time, whereas previous models failed this test entirely (0% NER).

Key Insights and Ideas

  • Time-Domain is King: By working directly with raw waveforms, X-TaSNet avoids the artifacts and quality loss associated with Short-Term Fourier Transforms (STFT). This shift alone accounts for a massive jump in audio quality.
  • Negative Constraints Matter: The "Aha!" moment here is that to find the right voice, the model must be explicitly taught what the wrong voices look like. The "Loss on Distortion" forces the model to understand the structure of the background noise, not just the target.
  • Robustness = Knowing when to quit: Most extraction models are "forced-choice"—they will always return something. By introducing SPIT, the authors moved toward a "triggerable" system that only activates when the correct speaker is detected.

Limitations and Future Work

Despite the progress, the authors note a few areas for improvement:

  • Absence Reliability: While 72.4% accuracy in detecting an absent speaker is a breakthrough, it isn't yet high enough for mission-critical applications.
  • Noise Diversity: The current study focuses primarily on "speech-on-speech" interference. Future versions should be tested against more complex backgrounds, such as music or heavy environmental noise.
  • Multi-Speaker Extraction: The current design is optimized for extracting one person; extending this to robustly handle multiple specific targets simultaneously remains a future goal.

Summary updated

Abstract

Extracting the speech of a target speaker from mixed audios, based on a reference speech from the target speaker, is a challenging yet powerful technology in speech processing. Recent studies of speaker-independent speech separation, such as TasNet, have shown promising results by applying deep neural networks over the time-domain waveform. Such separation neural network does not directly generate reliable and accurate output when target speakers are specified, because of the necessary prior on the number of speakers and the lack of robustness when dealing with audios with absent speakers. In this paper, we break these limitations by introducing a new speaker-aware speech masking method, called X-TaSNet. Our proposal adopts new strategies, including a distortion-based loss and corresponding alternating training scheme, to better address the robustness issue. X-TaSNet significantly enhances the extracted speech quality, doubling SDRi and SI-SNRi of the output speech audio over state-of-the-art voice filtering approach. X-TaSNet also improves the reliability of the results by improving the accuracy of speaker identity in the output audio to 95.4%, such that it returns silent audios in most cases when the target speaker is absent. These results demonstrate X-TaSNet moves one solid step towards more practical applications of speaker extraction technology.
BibTeX
@inproceedings{zhang2020x,
  author = {Zining Zhang and Bingsheng He and Zhenjie Zhang},
  title = {{X-TaSNet: Robust and Accurate Time-Domain Speaker Extraction Network}},
  booktitle = {Interspeech},
  year = {2020},
  doi = {10.21437/interspeech.2020-1706},
}
Figure