Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems

Jiajun Deng, Xurong Xie, Tianzi Wang, Mingyu Cui, Boyang Xue, Zengrui Jin, Guinan Li, Shujie Hu, Xunying Liu

Abstract

Speaker adaptation techniques provide a powerful solution to customise automatic speech recognition (ASR) systems for individual users. Practical application of unsupervised model-based speaker adaptation techniques to data intensive end-to-end ASR systems is hindered by the scarcity of speaker-level data and performance sensitivity to transcription errors. To address these issues, a set of compact and data efficient speaker-dependent (SD) parameter representations are used to facilitate both speaker adaptive training and test-time unsupervised speaker adaptation of state-of-the-art Conformer ASR systems. The sensitivity to supervision quality is reduced using a confidence score-based selection of the less erroneous subset of speaker-level adaptation data. Two lightweight confidence score estimation modules are proposed to produce more reliable confidence scores. The data sparsity issue, which is exacerbated by data selection, is addressed by modelling the SD parameter uncertainty using Bayesian learning. Experiments on the benchmark 300-hour Switchboard and the 233-hour AMI datasets suggest that the proposed confidence score-based adaptation schemes consistently outperformed the baseline speaker-independent (SI) Conformer model and conventional non-Bayesian, point estimate-based adaptation using no speaker data selection. Similar consistent performance improvements were retained after external Transformer and LSTM language model rescoring. In particular, on the 300-hour Switchboard corpus, statistically significant WER reductions of 1.0%, 1.3%, and 1.4% absolute (9.5%, 10.9%, and 11.3% relative) were obtained over the baseline SI Conformer on the NIST Hub5'00, RT02, and RT03 evaluation sets respectively. Similar WER reductions of 2.7% and 3.3% absolute (8.9% and 10.2% relative) were also obtained on the AMI development and evaluation sets.

Introduction

Modern End-to-End (E2E) Automatic Speech Recognition (ASR) systems, particularly those utilizing the Conformer architecture, have achieved state-of-the-art performance but remain highly data-hungry. A significant challenge arises when these systems encounter new speakers whose acoustic characteristics differ from the training set. While speaker adaptation offers a solution, it is often hindered by data sparsity (limited speaker-specific audio) and the supervision catch-22, where unsupervised models inadvertently learn from their own transcription errors, leading to compounded performance degradation.

Methods for Robust Adaptation

The proposed framework addresses these hurdles through a three-pronged strategy involving compact parameterization, neural confidence filtering, and Bayesian uncertainty modeling.

Compact Speaker-Dependent (SD) Parameters

Instead of updating the entire model, which would require massive datasets, the system inserts tiny, efficient modules: LHUC (Learning Hidden Unit Contributions), HUB (Hidden Unit Bias), and PAct (Parameterized Activation Functions). These act as targeted "knobs" to adjust the model to a specific voice with minimal data overhead.

Overview of the E2E ASR architecture with speaker-dependent transforms including LHUC, HUB, and PAct.

Confidence Score Estimation Modules (CEMs)

To solve the problem of learning from incorrect transcriptions, lightweight neural networks (CEMs) predict the reliability of each hypothesis. The system then selectively adapts using only the high-confidence subset of the data. By incorporating internal features like decoder embeddings and N-best hypothesis scores, these CEMs provide a more accurate "reality check" than raw output probabilities.

Bayesian Learning

To prevent overfitting to sparse speaker data, the authors utilize Bayesian variational inference. By modeling SD parameters as probability distributions rather than single point estimates, the system accounts for uncertainty, making the adaptation process significantly more stable.

Experimental Results

The researchers evaluated their methods on two major benchmarks: the Switchboard (300-hour) conversational telephone speech corpus and the AMI Meeting Corpus (233-hour).

Performance comparison of various adaptation methods on the Hub5'00 dataset for Transformer and Conformer models.

As shown in the performance tables, the proposed methods consistently outperformed baseline speaker-independent (SI) models. On Switchboard, the system achieved statistically significant Word Error Rate (WER) reductions of 1.0% to 1.4% absolute (approximately 9.5% to 11.3% relative). On the AMI corpus, gains were even more pronounced, with WER reductions of 2.7% and 3.3% absolute (8.9% to 10.2% relative).

Key Insights and Future Work

  • Early Adaptation is Key: Applying speaker-dependent transforms to the convolution subsampling module at the beginning of the encoder proved more effective than deeper layer updates, suggesting speaker variability is best captured early in the pipeline.
  • Quality Over Quantity: Experiments indicated that using the top 70-80% of "confident" data often yielded better results than using 100% of the data, highlighting the necessity of pruning transcription errors.
  • Future Directions: While effective, the current multi-pass decoding process is computationally intensive. Future work will focus on rapid, on-the-fly adaptation to reduce latency for real-time applications and extending these techniques to other architectures like RNN-Transducers.

Summary updated

Abstract

Speaker adaptation techniques provide a powerful solution to customise automatic speech recognition (ASR) systems for individual users. Practical application of unsupervised model-based speaker adaptation techniques to data intensive end-to-end ASR systems is hindered by the scarcity of speaker-level data and performance sensitivity to transcription errors. To address these issues, a set of compact and data efficient speaker-dependent (SD) parameter representations are used to facilitate both speaker adaptive training and test-time unsupervised speaker adaptation of state-of-the-art Conformer ASR systems. The sensitivity to supervision quality is reduced using a confidence score-based selection of the less erroneous subset of speaker-level adaptation data. Two lightweight confidence score estimation modules are proposed to produce more reliable confidence scores. The data sparsity issue, which is exacerbated by data selection, is addressed by modelling the SD parameter uncertainty using Bayesian learning. Experiments on the benchmark 300-hour Switchboard and the 233-hour AMI datasets suggest that the proposed confidence score-based adaptation schemes consistently outperformed the baseline speaker-independent (SI) Conformer model and conventional non-Bayesian, point estimate-based adaptation using no speaker data selection. Similar consistent performance improvements were retained after external Transformer and LSTM language model rescoring. In particular, on the 300-hour Switchboard corpus, statistically significant WER reductions of 1.0%, 1.3%, and 1.4% absolute (9.5%, 10.9%, and 11.3% relative) were obtained over the baseline SI Conformer on the NIST Hub5'00, RT02, and RT03 evaluation sets respectively. Similar WER reductions of 2.7% and 3.3% absolute (8.9% and 10.2% relative) were also obtained on the AMI development and evaluation sets.
BibTeX
@article{deng2023confidence,
  author = {Jiajun Deng and Xurong Xie and Tianzi Wang and Mingyu Cui and Boyang Xue and Zengrui Jin and Guinan Li and Shujie Hu and Xunying Liu},
  title = {{Confidence Score Based Speaker Adaptation of Conformer Speech Recognition Systems}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2023},
  doi = {10.1109/taslp.2023.3250842},
}
Figure