TL;DR
This paper introduces a novel Bayesian learning framework for deep neural network (DNN) speaker adaptation. By modeling speaker-dependent parameter uncertainty, their approach significantly reduces word error rates, especially when adaptation data is scarce, outperforming traditional deterministic methods.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems often suffer from a performance drop when encountering speech from speakers different from those in the training data. This "mismatch" is largely due to individual speaker characteristics like accent and speaking style. Speaker adaptation techniques aim to mitigate this by adjusting the ASR model to a specific speaker. However, existing model-based adaptation methods often require substantial amounts of speaker-specific data to avoid overfitting and poor generalization. This paper tackles the challenge of effective speaker adaptation with very limited adaptation data by explicitly modeling the uncertainty in speaker-dependent parameters.
Methods
The core contribution is a full Bayesian learning framework for DNN speaker adaptation. Instead of learning fixed speaker-dependent (SD) parameters, the authors propose learning probabilistic latent variable posterior distributions for these parameters for each speaker. This allows for explicit modeling of parameter uncertainty.
They investigate this framework within three established model-based DNN adaptation techniques:
- Bayesian Learning of Hidden Unit Contributions (BLHUC): Adapting scaling vectors for hidden layer outputs.
- Bayesian Hidden Unit Bias Vectors (BHUB): Adapting bias vectors for hidden layer outputs.
- Bayesian Parameterized Activation Functions (BPAct): Adapting parameters of activation functions like PReLU.
The posterior distributions are efficiently estimated using a variational inference approach, further enhanced by Monte Carlo parameter sampling and re-parameterization techniques. This ensures computational efficiency comparable to deterministic methods.
Datasets
- Switchboard: A 300-hour corpus used for training and experiments. An augmented 900-hour version was also used, created by speed perturbation.
- NIST Hub5'00: Used for evaluation, consisting of Switchboard (SWBD) and CallHome (CHE) subsets.
- RT03: Used for evaluation, consisting of Fisher (FSH) and Switchboard (SWBD) subsets.
- HKUST Mandarin Conversational Telephone Speech Database: Used for experiments on a different language and domain.
- DementiaBank Pitt Elderly Speech Corpus: Used to evaluate adaptation on elderly speech.
Key Insights and Ideas
- Modeling Uncertainty is Key for Low-Data Adaptation: The most significant insight is that explicitly modeling the uncertainty in speaker-dependent parameters (rather than treating them as fixed point estimates) is crucial for robust adaptation, especially when only a few utterances are available.
- Bayesian Adaptation Outperforms Deterministic Methods: Across various configurations and datasets, the proposed Bayesian adaptation consistently yields lower word error rates compared to their deterministic counterparts.
- BLHUC is Particularly Robust: The BLHUC adaptation, especially when applied to multiple layers, demonstrates robustness and consistently outperforms other methods, highlighting the benefit of adapting scaling factors.
- Computational Efficiency is Achieved: Despite the Bayesian approach, the use of variational inference and re-parameterization allows for computational costs comparable to traditional methods, making it practical.
- Generalizability of the Framework: The Bayesian framework is shown to be applicable to different types of model-based adaptation techniques, suggesting its broad utility.
Key Results
- Significant WER Reductions: The Bayesian adaptation approaches consistently outperformed deterministic methods. For instance, using only the first five utterances for adaptation, up to 1.4% absolute (7.2% relative) WER reduction was achieved on the CallHome subset.
- State-of-the-Art Performance: The best proposed BLHUC adapted CNN-TDNN system achieved state-of-the-art performance with a 10.7% WER on the RT03 evaluation set.
- Robustness to Data Scarcity: The Bayesian methods showed significant improvements even with very limited adaptation data (e.g., 5 utterances), with improvements increasing as more data became available.
- Comparison to Other Regularization Methods: BLHUC adaptation consistently outperformed other regularization techniques like f-MLLR, KL divergence, and MAP adaptation.
Limitations and Future Work
- Non-Variational Inference: The paper primarily uses variational inference. Future work could explore non-variational inference methods for Bayesian adaptation.
- Computational Cost: While the paper demonstrates comparable efficiency, further optimization for extremely low-resource scenarios might be explored.
- Exploration of Other Architectures: While tested on TDNN and CNN-TDNN, exploring the Bayesian framework with other advanced architectures like Transformers could be beneficial.
Summary updated