Profile-Error-Tolerant Target-Speaker Voice Activity Detection

Dongmei Wang, Xiong Xiao, Naoyuki Kanda, Midia Yousefi, Takuya Yoshioka, Jian Wu

In the world of speech processing, speaker diarization—the task of determining "who spoke when"—is a foundational challenge. While recent deep learning models have made massive strides, they often rely on a "first-pass" guess that can be prone to errors. A new paper from Microsoft researchers introduces PET-TSVAD (Profile-Error-Tolerant Target-Speaker Voice Activity Detection), a system designed to keep working even when that initial guess is wrong.

TL;DR

PET-TSVAD is an evolution of the state-of-the-art Target-Speaker Voice Activity Detection (TS-VAD) model that handles errors in initial speaker discovery. By introducing "pseudo-speaker profiles" and training on intentionally imperfect data, the model can detect speakers missed by traditional clustering methods, significantly reducing diarization error rates on challenging datasets like DIHARD-I.

What Problem Does This Paper Address?

Standard TS-VAD models are powerful because they don't just look at the audio; they look for specific speaker "profiles" (mathematical signatures of a voice) within that audio. However, these profiles are usually generated by a traditional clustering algorithm in a "first pass."

This creates a "garbage in, garbage out" vulnerability:

  1. Missed Speakers: If the first pass fails to detect a speaker, the TS-VAD model has no profile for them and will never "look" for them.
  2. Split/Merged Speakers: If the first pass accidentally splits one person into two profiles (or merges two people into one), the TS-VAD model gets confused, leading to high speaker confusion errors.
  3. Train-Test Mismatch: Most models are trained using "oracle" (perfect) profiles but are tested on "noisy" profiles from real-world clustering, leading to a performance drop.

Methods

The researchers built PET-TSVAD on top of a Transformer-based architecture to handle a variable number of speakers. They introduced three key innovations:

  • Pseudo-Speaker Profiles: The model is given a set of "blank" learnable embeddings (pseudo-profiles) alongside the profiles found in the first pass. These act as safety nets; if the model hears a voice that doesn't match any provided profile, it can assign that activity to a pseudo-profile slot.
  • Noisy Profile Training: Instead of training the model with perfect speaker data, the authors ran multiple different clustering algorithms (like AHC and NME-SC) on their training data. This forced the model to learn how to handle the types of errors it would actually encounter in the real world.
  • Permutation Invariant Training (PIT): Because the input profiles are no longer perfectly mapped to the output labels (due to clustering errors), the authors used PIT. This allows the model to find the best possible match between its predictions and the ground truth during training, regardless of the order.

Datasets

The model was evaluated on two major benchmarks:

  • VoxConverse: A dataset derived from YouTube videos, featuring multi-speaker conversations with varying overlap.
  • DIHARD-I: A notoriously difficult dataset containing clinical interviews, restaurant conversations, and meetings with high levels of noise and overlapping speech.
  • VoxCeleb1 & 2: Used to simulate 21,000 hours of multi-talker audio for large-scale pre-training.

Key Insights and Ideas

  • The "Aha" Moment for Pseudo-Profiles: The researchers found that applying clustering to the pseudo-profiles across different audio chunks actually degraded performance. It turns out the model is better off treating each chunk's missing speakers independently rather than trying to force a global identity on these "ghost" speakers.
  • Diversity is Key: Using a mix of different clustering thresholds during training was more effective than using a single "best" clustering method. This diversity made the model more robust to various types of profile contamination.
  • Reducing Confusion: The biggest performance gain didn't come from just finding more speech, but from significantly reducing "speaker confusion"—the error of attributing the right speech to the wrong person.

Key Results

PET-TSVAD consistently outperformed the baseline Transformer-TS-VAD:

  • DIHARD-I: The model achieved a 7.4% relative reduction in Diarization Error Rate (DER), dropping from 27.94% to 25.88%.
  • VoxConverse: The DER improved from 4.46% to 4.35%.
  • The results showed that even when the initial clustering was quite poor (high "miss" or "false alarm" rates), PET-TSVAD was able to "clean up" the results and recover the correct speaker boundaries.

Limitations and Future Work

While PET-TSVAD is more robust, it still processes audio in chunks (e.g., 60 or 120 seconds). The authors noted that a speaker might appear in different "pseudo-profile" slots in different chunks, which complicates long-form diarization. Future work may involve better ways to stitch these pseudo-speaker detections together across an entire hour-long recording without introducing new errors. Additionally, the computational cost of Permutation Invariant Training (PIT) increases as the number of speakers grows, which could be a bottleneck for very large meetings.

Summary updated

Abstract

Target-Speaker Voice Activity Detection (TS-VAD) utilizes a set of speaker profiles alongside an input audio signal to perform speaker diarization. While its superiority over conventional methods has been demonstrated, the method can suffer from errors in speaker profiles, as those profiles are typically obtained by running a traditional clustering-based diarization method over the input signal. This paper proposes an extension to TS-VAD, called Profile-Error-Tolerant TS- VAD (PETTSVAD), which is robust to such speaker profile errors. This is achieved by employing transformer-based TS-VAD that can handle a variable number of speakers and further introducing a set of additional pseudo-speaker profiles to handle speakers undetected during the first pass diarization. During training, we use speaker profiles estimated by multiple different clustering algorithms to reduce the mismatch between the training and testing conditions regarding speaker profiles. Experimental results show that PET-TSVAD consistently outperforms the existing TS-VAD method on both the VoxConverse and DIHARD-I datasets.
BibTeX
@inproceedings{wang2024profile,
  author = {Dongmei Wang and Xiong Xiao and Naoyuki Kanda and Midia Yousefi and Takuya Yoshioka and Jian Wu},
  title = {{Profile-Error-Tolerant Target-Speaker Voice Activity Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=TgHCevgMch},
}

Loading related content...

Figure