Streaming Speaker Change Detection and Gender Classification for Transducer-Based Multi-Talker Speech Translation

Peidong Wang, Naoyuki Kanda, Jian Xue, Jinyu Li, Xiaofei Wang, Aswin Shanmugam Subramanian, Jun-Kun Chen, Sunit Sivasankaran, Xiong Xiao, Yong ZhaoarXiv 2025 · DOI · OpenReview

TL;DR

This paper tackles the challenge of real-time speech translation for conversations with multiple speakers. They propose a novel approach that integrates speaker change detection and gender classification directly into a streaming end-to-end speech translation model using speaker embeddings, achieving high accuracy for both tasks.

What Problem Does This Paper Address?

Translating spoken conversations with multiple speakers in real-time is a complex task. Beyond simply transcribing and translating speech accurately and with low latency, it's crucial to know who is speaking and when they change. This information is vital for downstream applications like text-to-speech (TTS) systems. For instance, zero-shot TTS models rely on accurate audio prompts to generate realistic speech, and conventional TTS systems use speaker profiles, which often require gender information for selection. Existing methods for speaker diarization (identifying speakers) and speech translation are often offline or don't integrate seamlessly, leading to potential error accumulation or latency issues. This paper aims to bridge this gap by developing a unified, streaming system for multi-talker speech translation that also handles speaker change detection and gender classification.

Methods

The core of the proposed method lies in integrating speaker embeddings into a transducer-based streaming end-to-end speech translation (ST) model. Specifically, they leverage the t-vector technique, which generates token-level speaker embeddings.

Here's a breakdown:

  • Transducer-based Streaming ST: They use a Transformer transducer architecture, which is well-suited for streaming scenarios due to its ability to process audio in chunks with a limited reception field. This allows for low-latency translation.
  • Multilingual Capability: The model is designed to handle multiple target languages by using language identification (LID) tokens.
  • t-vector for Speaker Information: A t-vector module is added to the pre-trained ST model. This module consists of a speaker encoder and a speaker decoder. The speaker encoder extracts speaker-specific information from the ST encoder's layers.
  • Speaker Change Detection: This is achieved by calculating the cosine similarity between adjacent t-vectors. If the similarity falls below a predefined threshold, a speaker change is detected and an <SC> token is inserted into the translation output. This method is data-efficient and doesn't require retraining the ST model.
  • Gender Classification: For each output token, the cosine similarity between its t-vector and pre-defined male and female speaker profiles (also represented by d-vectors) is calculated. The gender with the highest similarity score is assigned to the token.

Datasets

  • Training Data:
    • ST Model: Approximately 75K hours of English audio from various anonymized sources.
  • t-vector Model: Voxceleb (Nagrani et al., 2017) was used for training the t-vector module.
  • Test Data:
    • Speaker Change Detection: 5 real recorded long conversational audios, segmented and annotated by human annotators for speaker change points. This resulted in 688 samples, each containing one speaker change, with an average duration of about 5 seconds.
  • Gender Classification: The test set of Voxceleb-1 was used, with speaker profiles from Voxceleb-1 and Voxceleb-2 divided into male and female categories. This resulted in 4874 utterances.

Key Insights and Ideas

  • Unified Streaming System: The most significant insight is the successful integration of speaker change detection and gender classification directly into a streaming ST pipeline. This avoids the need for separate, potentially offline, diarization systems.
  • Data Efficiency of t-vectors: The t-vector approach for speaker change detection is presented as data-efficient, treating it as a speaker identification (SID) generation task rather than requiring large amounts of real-world conversational data with explicit speaker change annotations.
  • Preservation of ST Performance: By fixing the ST model parameters during t-vector training, the authors ensure that the translation quality is not negatively impacted by the addition of speaker-related tasks.
  • Token-Level Granularity: Performing speaker change detection and gender classification at the token level allows for fine-grained analysis and integration with the translated text output.

Key Results

  • Speaker Change Detection: The proposed method achieved F₁ scores above 0.6 across different thresholds, comparable to offline models like Whisper and EEND, but with the advantage of streaming and direct integration with translated text. The method is noted to be robust to noise and silence, though these can still cause some errors.
  • Gender Classification: The token-level gender classification accuracy was very high, reaching 0.989 across different languages, indicating the robustness of the t-vector approach for this task.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the test data for speaker change detection contained noise and silence, which are challenging for the model. Most incorrect detections occurred at these points.
  • While comparable to offline models, the F₁ scores for speaker change detection are still below 0.8, suggesting room for improvement.
  • The paper mentions that the exact timestamps of speaker changes can vary slightly between different translation outputs due to the non-deterministic nature of the ST model.
  • Future Work:
    • The authors suggest that their method is robust to output languages and can be applied to many-to-EN translation, implying future work could explore broader multilingual scenarios.
  • The focus on streaming speaker change detection and gender classification is essential for subsequent TTS modules, hinting at further integration and optimization with TTS systems.

Summary updated

Abstract

Streaming multi-talker speech translation is a task that involves not only generating accurate and fluent translations with low latency but also recognizing when a speaker change occurs and what the speaker's gender is. Speaker change information can be used to create audio prompts for a zero-shot text-to-speech system, and gender can help to select speaker profiles in a conventional text-to-speech model. We propose to tackle streaming speaker change detection and gender classification by incorporating speaker embeddings into a transducer-based streaming end-to-end speech translation model. Our experiments demonstrate that the proposed methods can achieve high accuracy for both speaker change detection and gender classification.
BibTeX
@misc{wang2025streaming,
  author = {Peidong Wang and Naoyuki Kanda and Jian Xue and Jinyu Li and Xiaofei Wang and Aswin Shanmugam Subramanian and Jun-Kun Chen and Sunit Sivasankaran and Xiong Xiao and Yong Zhao},
  title = {{Streaming Speaker Change Detection and Gender Classification for Transducer-Based Multi-Talker Speech Translation}},
  year = {2025},
  eprint = {2502.02683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.02683},
}
Figure