Profile-Error-Tolerant Target-Speaker Voice Activity Detection

Dongmei Wang, Xiong Xiao, Naoyuki Kanda, Midia Yousefi, Takuya Yoshioka, Jian Wu

Abstract

Target-Speaker Voice Activity Detection (TS-VAD) utilizes a set of speaker profiles alongside an input audio signal to perform speaker diarization. While its superiority over conventional methods has been demonstrated, the method can suffer from errors in speaker profiles, as those profiles are typically obtained by running a traditional clustering-based diarization method over the input signal. This paper proposes an extension to TS-VAD, called Profile-Error-Tolerant TS-VAD (PET-TSVAD), which is robust to such speaker profile errors. This is achieved by employing transformer-based TS-VAD that can handle a variable number of speakers and further introducing a set of additional pseudo-speaker profiles to handle speakers undetected during the first pass diarization. During training, we use speaker profiles estimated by multiple different clustering algorithms to reduce the mismatch between the training and testing conditions regarding speaker profiles. Experimental results show that PET-TSVAD consistently outperforms the existing TS-VAD method on both the VoxConverse and DIHARD-I datasets.

Introduction

Speaker diarization—the task of determining "who spoke when"—is a foundational challenge in speech processing. While recent deep learning models have made massive strides, they often rely on a "first-pass" guess that can be prone to errors. Standard Target-Speaker Voice Activity Detection (TS-VAD) models are powerful because they look for specific speaker "profiles" (mathematical signatures of a voice) within audio. However, these profiles are usually generated by a traditional clustering algorithm in an initial pass, creating a "garbage in, garbage out" vulnerability. If the first pass fails to detect a speaker or incorrectly merges identities, the TS-VAD model lacks the necessary profiles to perform accurately.

Methods and Architecture

PET-TSVAD is an evolution of the state-of-the-art TS-VAD model designed to handle errors in initial speaker discovery. The researchers built PET-TSVAD on top of a Transformer-based architecture to handle a variable number of speakers, introducing three key innovations:

  • Pseudo-Speaker Profiles: The model is provided with a set of "blank" learnable embeddings (pseudo-profiles) alongside the profiles found in the first pass. These act as safety nets; if the model detects a voice that does not match any provided profile, it can assign that activity to a pseudo-profile slot.
  • Noisy Profile Training: Instead of training with perfect "oracle" profiles, the authors used multiple clustering algorithms (like AHC and NME-SC) on training data. This forces the model to learn how to handle the types of errors it will encounter in real-world scenarios.
  • Permutation Invariant Training (PIT): Because input profiles may not perfectly map to output labels due to clustering errors, PIT allows the model to find the best possible match between predictions and ground truth during training.
The PET-TSVAD architecture incorporating pseudo-speaker profiles and a transformer-based detection backend.

Experimental Results

The model was evaluated on two major benchmarks: VoxConverse (YouTube-derived multi-speaker conversations) and DIHARD-I (a difficult dataset with high noise and overlapping speech). PET-TSVAD consistently outperformed the baseline Transformer-TS-VAD across both.

On the DIHARD-I dataset, the model achieved a 7.4% relative reduction in Diarization Error Rate (DER), dropping from 27.94% to 25.88%. On VoxConverse, the DER improved from 4.46% to 4.35%. The results demonstrate that even when initial clustering is poor, PET-TSVAD can "clean up" results and recover correct speaker boundaries, primarily by reducing "speaker confusion"—the error of attributing speech to the wrong person.

Performance comparison on the DIHARD-I dataset, showing PET-TSVAD-2 (S6) achieving the lowest DER.

Conclusion and Future Work

PET-TSVAD significantly improves robustness to profile errors in speaker diarization. However, the model currently processes audio in chunks (e.g., 60-120 seconds), which can lead to a speaker appearing in different "pseudo-profile" slots across different segments. Future work will focus on better methods for stitching these pseudo-speaker detections together across long-form recordings and addressing the computational scaling of Permutation Invariant Training for very large numbers of speakers.

Summary updated

Abstract

Target-Speaker Voice Activity Detection (TS-VAD) utilizes a set of speaker profiles alongside an input audio signal to perform speaker diarization. While its superiority over conventional methods has been demonstrated, the method can suffer from errors in speaker profiles, as those profiles are typically obtained by running a traditional clustering-based diarization method over the input signal. This paper proposes an extension to TS-VAD, called Profile-Error-Tolerant TS-VAD (PET-TSVAD), which is robust to such speaker profile errors. This is achieved by employing transformer-based TS-VAD that can handle a variable number of speakers and further introducing a set of additional pseudo-speaker profiles to handle speakers undetected during the first pass diarization. During training, we use speaker profiles estimated by multiple different clustering algorithms to reduce the mismatch between the training and testing conditions regarding speaker profiles. Experimental results show that PET-TSVAD consistently outperforms the existing TS-VAD method on both the VoxConverse and DIHARD-I datasets.
BibTeX
@inproceedings{wang2024profile,
  author = {Dongmei Wang and Xiong Xiao and Naoyuki Kanda and Midia Yousefi and Takuya Yoshioka and Jian Wu},
  title = {{Profile-Error-Tolerant Target-Speaker Voice Activity Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=TgHCevgMch},
}
Figure