Abstract
Recently, an end-to-end speaker-attributed automatic speech recognition (E2E SA-ASR) model was proposed as a joint model of speaker counting, speech recognition and speaker identification for monaural overlapped speech. In the previous study, the model parameters were trained based on the speaker-attributed maximum mutual information (SA-MMI) criterion, with which the joint posterior probability for multi-talker transcription and speaker identification are maximized over training data. Although SA-MMI training showed promising results for overlapped speech consisting of various numbers of speakers, the training criterion was not directly linked to the final evaluation metric, i.e., speaker-attributed word error rate (SA-WER). In this paper, we propose a speaker-attributed minimum Bayes risk (SA-MBR) training method where the parameters are trained to directly minimize the expected SA-WER over the training data. Experiments using the LibriSpeech corpus show that the proposed SA-MBR training reduces the SA-WER by 9.0 % relative compared with the SA-MMI-trained model.
Introduction
Transcribing "who said what" in multi-talker environments, such as business meetings, remains a significant challenge in speech processing. This task, Speaker-Attributed Automatic Speech Recognition (SA-ASR), requires a system to simultaneously perform speaker counting, speech separation, recognition, and identification. While recent end-to-end (E2E) models have integrated these components into a single neural network, they have traditionally relied on the Maximum Mutual Information (SA-MMI) criterion for training. SA-MMI focuses on maximizing the probability of the correct transcript, which does not always align with the ultimate goal of minimizing the Speaker-Attributed Word Error Rate (SA-WER). This misalignment is particularly problematic in scenarios with high speaker overlap.
Methods
We propose Speaker-Attributed Minimum Bayes Risk (SA-MBR) training to bridge the gap between the training objective and the evaluation metric.
Model Architecture
The E2E SA-ASR model consists of two primary interdependent blocks:
- ASR Block: An encoder-decoder architecture that generates text transcriptions.
- Speaker Identification (SID) Block: This block utilizes speaker profiles (d-vectors) to identify individuals associated with each recognized word.
SA-MBR Training Objective
Unlike standard cross-entropy or MMI losses, SA-MBR calculates the expected error across the $N$-best hypotheses generated by the model. By evaluating the SA-WER for the top $N$ candidates against the ground truth, the model parameters are updated to favor hypotheses with lower expected errors. This represents the first application of MBR to a joint framework handling both transcription and speaker identification.
Length Normalization
A critical finding in our study was the necessity of length normalization. Normalizing posterior probabilities by sequence length was essential to mitigate biases toward specific sequence lengths, which is a common issue in sequence-to-sequence MBR training for multi-speaker data.
Experimental Setup
The proposed method was evaluated using the LibriSpeech corpus (960 hours). We created LibriSpeechMix, a simulated multi-speaker dataset featuring 1 to 3 speakers with random delays and an average signal-to-interference ratio of 0 dB. Speaker identification relied on 128-dimensional d-vectors extracted from a model pre-trained on the VoxCeleb corpus. During testing, the model was provided with a speaker inventory of 8 profiles, including the target speakers and distractors.
Key Results
The SA-MBR method demonstrated consistent improvements over the SA-MMI baseline across all metrics.
As shown in the results, SA-MBR achieved a 9.0% relative reduction in total SA-WER, decreasing from 15.6% to 14.2%. The performance gains were most significant in complex 3-speaker scenarios, where SA-WER dropped from 23.1% to 20.7%. Additionally, the total Speaker Error Rate (SER) improved from 6.0% to 5.3%.
Discussion and Insights
Improved Speaker Counting
An unexpected benefit of SA-MBR training was the enhancement of speaker counting accuracy. In 3-speaker cases, accuracy improved from 74.73% to 78.32%.
We hypothesize that because missing a "speaker change" tag significantly increases SA-WER, the MBR objective penalizes such errors more severely than the MMI criterion.
Limitations and Future Work
While promising, the current system depends on pre-existing speaker profiles (d-vectors), which may not be available in all real-world scenarios. Furthermore, while LibriSpeechMix is a robust benchmark, it may not fully capture the acoustic complexities of natural conversations, such as reverberation. Future work will explore integrating external language models and data augmentation to further reduce error rates.
Conclusion
The introduction of SA-MBR training for E2E SA-ASR models provides a more direct path to optimizing for the metrics that matter most in multi-talker transcription. By minimizing expected SA-WER, the model achieves superior performance in both recognition and speaker attribution, particularly in challenging overlapped speech conditions.
Summary updated