Speaker Change Detection For Transformer Transducer ASR

Jian Wu, Zhuo Chen, Min Hu, Xiong Xiao, Jinyu Li

Introduction

Speaker Change Detection (SCD) is a critical component in modern speech recognition systems, acting as the "punctuation" of the speaker world. It enables the generation of readable transcripts by identifying when a new speaker begins, which is essential for downstream tasks like speaker diarization. However, integrating SCD into streaming Automatic Speech Recognition (ASR) systems presents a challenge. Existing solutions typically use either time-based detection, which often misaligns with word boundaries, or joint word-based detection, which frequently leads to a "tug-of-war" during training where optimizing for SCD degrades the Word Error Rate (WER).

Proposed Architecture

To address these issues, researchers at Microsoft proposed a decoupled framework where a pre-existing Transformer Transducer (TT-ASR) serves as a feature extractor for a specialized SCD module. This modular approach allows both tasks to reach optimum performance without interference.

Architecture of the proposed SCD-ENC and SCD-DEC schemes integrated with a TT-ASR backbone.

The framework explores two primary implementations:

  • SCD-ENC (Alignment-based): This scheme utilizes forced alignment to determine precise word boundaries. It averages acoustic features from the ASR encoder within these windows and combines them with text embeddings to predict speaker changes.
  • SCD-DEC (Decoder-based): Designed to reduce computational overhead, this version employs a Transformer Decoder with an attention mechanism to simultaneously process ASR encoder features and recognized tokens.

A key innovation is the use of chunk-wise inference, allowing the ASR to maintain low latency (160ms) while the SCD module utilizes a slightly larger context window for more accurate decision-making.

Key Insights

  • The Shallow Layer Advantage: Experiments revealed that features from intermediate layers of the ASR encoder (e.g., layer 4 of 18) are superior for SCD compared to the final layer. While deeper layers are optimized for linguistic content, shallower layers retain more acoustic information relevant to speaker identity.
  • Decoupling for Stability: By treating SCD as a separate module, the team could apply Focal Loss to address class imbalance (as speaker changes are rare) without impacting the ASR training objective.
  • Contextual Look-ahead: SCD performance scales significantly with increased look-ahead context. Because the module is decoupled, this extra latency for speaker labels does not affect the near-instantaneous delivery of text labels.

Experimental Results

The proposed methods were evaluated on LibriCSS (LS-ST) and call center data (MS-CC). The results demonstrate that decoupling SCD from the ASR backbone provides superior performance over joint training baselines.

Comparison of WER and F1 scores across different methods, highlighting the performance gains of SCD-ENC and SCD-DEC without WER degradation.

As shown in the results, the SCD-ENC model achieved a peak F1 score of 91.3% on LibriCSS, a substantial improvement over the 74.5% achieved by the best joint baseline. Crucially, this approach maintained a stable WER of 8.65%, whereas joint models often suffer from accuracy degradation. Even on challenging 8kHz call center data, the models remained robust, maintaining F1 scores above 81%.

Conclusion and Future Work

The modular SCD framework successfully bridges the gap between high-accuracy transcription and precise speaker change detection. While SCD-ENC offers the highest accuracy, it requires forced alignment; conversely, SCD-DEC is more computationally efficient but slightly less accurate due to word emission latency. Future research will focus on reducing computational costs, further minimizing emission latency, and extending the system to handle complex multi-talker scenarios involving overlapping speech.

Summary updated

BibTeX
@inproceedings{wu2023speaker,
  author = {Jian Wu and Zhuo Chen and Min Hu and Xiong Xiao and Jinyu Li},
  title = {{Speaker Change Detection For Transformer Transducer ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096361},
}
Figure