SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR

ShiLiang Zhang, Zhiyun Fan, Linhao Dong, Jun Zhang, Lu Lu, Zejun MaICASSP 2024 · DOI

TL;DR

This paper introduces SA-SOT, a novel approach to multi-talker Automatic Speech Recognition (ASR) that tackles the "semantic confusion" problem in existing serialized output training (SOT) methods. By integrating speaker embeddings into the ASR decoder and using a speaker similarity matrix to refine attention, SA-SOT significantly improves transcription accuracy in scenarios with overlapping speech, achieving state-of-the-art results.

What Problem Does This Paper Address?

Multi-talker ASR is crucial for understanding conversations in environments like meetings where multiple people speak simultaneously. Traditional single-talker ASR systems struggle with this. While serialized output training (SOT) has shown promise by simplifying the model architecture and training process for multi-talker ASR, it faces a significant challenge: the autoregressive decoder, which predicts tokens sequentially, gets confused when speaker turns change rapidly. The decoder needs to distinguish between relevant context from the current speaker and irrelevant interjections from other speakers, a task that becomes difficult when all speech is merged into a single sequence with special speaker-change tokens (<cc>). This "semantic confusion" hinders accurate transcription, especially in token-level SOT (t-SOT).

Methods

The proposed SA-SOT method enhances the t-SOT framework with three key innovations:

  • Speaker Embedding Fusion (SEF): A separate speaker branch extracts speaker embeddings. These embeddings are then fused into the ASR decoder's input, providing explicit speaker information to aid in distinguishing between speakers.
  • Speaker-Aware Training (SAT): A "masked t-SOT" label is introduced. This label uses a special <mask> token to explicitly hide tokens belonging to other speakers for a given speaker's sequence. This auxiliary loss encourages the model to focus on the correct speaker's speech.
  • Speaker-Aware Attention (SAA): A speaker similarity matrix is computed based on the token-level speaker embeddings. This matrix is used to reweight the self-attention mechanism within the ASR decoder. The goal is to strengthen the attention paid to tokens from the same speaker while reducing attention to tokens from different speakers.

The overall framework involves joint training of the ASR and speaker branches, with a combined loss function that includes standard ASR losses (CTC, CE, quantity loss) and an AMsoftmax loss for speaker discrimination, along with the new SAT loss.

Datasets

The experiments were conducted on:

  • Librispeech: The 960-hour training data was used to simulate multi-talker training data. The test-clean and test-other sets were used to create multi-talker test sets (test-clean-2mix, test-other-2mix) by overlapping speech from two speakers.
  • LibrispeechMix: This is a pre-existing test set containing two speakers per speech, used for evaluation.

The authors also evaluated on the single-talker test-clean set of Librispeech to ensure performance wasn't degraded on simpler tasks.

Key Insights and Ideas

  • Explicit speaker information is crucial for decoder context: Simply marking speaker changes with <cc> isn't enough. Providing direct speaker embeddings to the decoder significantly helps it disambiguate overlapping speech.
  • Masking irrelevant speaker context aids learning: The masked t-SOT label acts as a strong signal, forcing the model to learn speaker-specific contexts more effectively.
  • Attention can be speaker-aware: Modifying the self-attention mechanism with speaker similarity scores allows the model to dynamically adjust its focus, prioritizing intra-speaker context over inter-speaker context.
  • SA-SOT improves multi-talker ASR without harming single-talker performance: A key finding is that the proposed method enhances performance on challenging multi-talker scenarios while maintaining or even slightly improving performance on standard single-talker tasks.

Key Results

  • Significant cpWER reduction: SA-SOT achieved relative cpWER reductions ranging from 12.75% to 22.03% on the multi-talker test sets (test-clean-2mix and test-other-2mix) compared to the baseline CIF t-SOT model.
  • State-of-the-art performance: With extensive training, SA-SOT achieved a cpWER of 3.41% on the LibrispeechMix test set, setting a new state-of-the-art.
  • Impact of components: Speaker embedding fusion (SEF), speaker-aware training (SAT), and speaker-aware attention (SAA) all contributed to performance gains, with SEF and SAT showing the most significant improvements.
  • Encoder downsampling trade-off: While higher frame rates (1/2 or 1/4 downsampling) benefited multi-talker performance by providing more detail, they slightly degraded single-talker performance due to increased sequence length complexity.

Limitations and Future Work

  • Computational cost of speaker similarity matrix: While effective, computing the N x N similarity matrix adds computational overhead.
  • Extensive training: Achieving state-of-the-art results required extensive training, suggesting potential for further optimization or more efficient training strategies.
  • Future work: The authors suggest exploring more advanced speaker embedding fusion techniques and investigating the impact of their method on real-world, unsegmented meeting recordings. The paper also hints at the potential for extending this approach to scenarios with more than two speakers.

Summary updated

BibTeX
@inproceedings{zhang2024sa,
  author = {ShiLiang Zhang and Zhiyun Fan and Linhao Dong and Jun Zhang and Lu Lu and Zejun Ma},
  title = {{SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446059},
}
Figure