Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASR

Naoyuki Kanda, Zhong Meng, Liang Lu, Yashesh Gaur, Xiaofei Wang, Zhuo Chen, Takuya Yoshioka

Abstract

Recently, an end-to-end speaker-attributed automatic speech recognition (E2E SA-ASR) model was proposed as a joint model of speaker counting, speech recognition and speaker identification for monaural overlapped speech. In the previous study, the model parameters were trained based on the speaker-attributed maximum mutual information (SA-MMI) criterion, with which the joint posterior probability for multi-talker transcription and speaker identification are maximized over training data. Although SA-MMI training showed promising results for overlapped speech consisting of various numbers of speakers, the training criterion was not directly linked to the final evaluation metric, i.e., speaker-attributed word error rate (SA-WER). In this paper, we propose a speaker-attributed minimum Bayes risk (SA-MBR) training method where the parameters are trained to directly minimize the expected SA-WER over the training data. Experiments using the LibriSpeech corpus show that the proposed SA-MBR training reduces the SA-WER by 9.0 % relative compared with the SA-MMI-trained model.

Introduction

Transcribing "who said what" in multi-talker environments, such as business meetings, remains a significant challenge in speech processing. This task, Speaker-Attributed Automatic Speech Recognition (SA-ASR), requires a system to simultaneously perform speaker counting, speech separation, recognition, and identification. While recent end-to-end (E2E) models have integrated these components into a single neural network, they have traditionally relied on the Maximum Mutual Information (SA-MMI) criterion for training. SA-MMI focuses on maximizing the probability of the correct transcript, which does not always align with the ultimate goal of minimizing the Speaker-Attributed Word Error Rate (SA-WER). This misalignment is particularly problematic in scenarios with high speaker overlap.

Methods

We propose Speaker-Attributed Minimum Bayes Risk (SA-MBR) training to bridge the gap between the training objective and the evaluation metric.

Model Architecture

The E2E SA-ASR model consists of two primary interdependent blocks:

  1. ASR Block: An encoder-decoder architecture that generates text transcriptions.
  2. Speaker Identification (SID) Block: This block utilizes speaker profiles (d-vectors) to identify individuals associated with each recognized word.

SA-MBR Training Objective

Unlike standard cross-entropy or MMI losses, SA-MBR calculates the expected error across the $N$-best hypotheses generated by the model. By evaluating the SA-WER for the top $N$ candidates against the ground truth, the model parameters are updated to favor hypotheses with lower expected errors. This represents the first application of MBR to a joint framework handling both transcription and speaker identification.

Length Normalization

A critical finding in our study was the necessity of length normalization. Normalizing posterior probabilities by sequence length was essential to mitigate biases toward specific sequence lengths, which is a common issue in sequence-to-sequence MBR training for multi-speaker data.

Experimental Setup

The proposed method was evaluated using the LibriSpeech corpus (960 hours). We created LibriSpeechMix, a simulated multi-speaker dataset featuring 1 to 3 speakers with random delays and an average signal-to-interference ratio of 0 dB. Speaker identification relied on 128-dimensional d-vectors extracted from a model pre-trained on the VoxCeleb corpus. During testing, the model was provided with a speaker inventory of 8 profiles, including the target speakers and distractors.

Key Results

The SA-MBR method demonstrated consistent improvements over the SA-MMI baseline across all metrics.

Table 2: Comparison of SER, WER, and SA-WER (%) for SA-MMI and SA-MBR models across different speaker counts.

As shown in the results, SA-MBR achieved a 9.0% relative reduction in total SA-WER, decreasing from 15.6% to 14.2%. The performance gains were most significant in complex 3-speaker scenarios, where SA-WER dropped from 23.1% to 20.7%. Additionally, the total Speaker Error Rate (SER) improved from 6.0% to 5.3%.

Discussion and Insights

Improved Speaker Counting

An unexpected benefit of SA-MBR training was the enhancement of speaker counting accuracy. In 3-speaker cases, accuracy improved from 74.73% to 78.32%.

Table 4: Speaker counting accuracy for SA-MMI and SA-MBR models.

We hypothesize that because missing a "speaker change" tag significantly increases SA-WER, the MBR objective penalizes such errors more severely than the MMI criterion.

Limitations and Future Work

While promising, the current system depends on pre-existing speaker profiles (d-vectors), which may not be available in all real-world scenarios. Furthermore, while LibriSpeechMix is a robust benchmark, it may not fully capture the acoustic complexities of natural conversations, such as reverberation. Future work will explore integrating external language models and data augmentation to further reduce error rates.

Conclusion

The introduction of SA-MBR training for E2E SA-ASR models provides a more direct path to optimizing for the metrics that matter most in multi-talker transcription. By minimizing expected SA-WER, the model achieves superior performance in both recognition and speaker attribution, particularly in challenging overlapped speech conditions.

Summary updated

Abstract

Recently, an end-to-end speaker-attributed automatic speech recognition (E2E SA-ASR) model was proposed as a joint model of speaker counting, speech recognition and speaker identification for monaural overlapped speech. In the previous study, the model parameters were trained based on the speaker-attributed maximum mutual information (SA-MMI) criterion, with which the joint posterior probability for multi-talker transcription and speaker identification are maximized over training data. Although SA-MMI training showed promising results for overlapped speech consisting of various numbers of speakers, the training criterion was not directly linked to the final evaluation metric, i.e., speaker-attributed word error rate (SA-WER). In this paper, we propose a speaker-attributed minimum Bayes risk (SA-MBR) training method where the parameters are trained to directly minimize the expected SA-WER over the training data. Experiments using the LibriSpeech corpus show that the proposed SA-MBR training reduces the SA-WER by 9.0 % relative compared with the SA-MMI-trained model.
BibTeX
@inproceedings{kanda2021minimum,
  author = {Naoyuki Kanda and Zhong Meng and Liang Lu and Yashesh Gaur and Xiaofei Wang and Zhuo Chen and Takuya Yoshioka},
  title = {{Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=wxUJ3ueFrhS},
}
Figure