Shrinkage model adaptation in automatic speech recognition

Jinyu Li, Yu Tsao, Chin-Hui Lee

Introduction

In automatic speech recognition (ASR), model adaptation is crucial for tailoring general acoustic models to specific speakers or environments. However, standard techniques like Maximum Likelihood Linear Regression (MLLR) and Constrained MLLR (CMLLR) often struggle when adaptation data is extremely limited—such as a single spoken sentence. In these "data-starved" scenarios, the estimation of transformation parameters becomes unreliable, leading to overfitting. This paper, presented at INTERSPEECH 2010, proposes a robust solution by incorporating "shrinkage" techniques from classical statistics into the adaptation process.

The Problem of Data Insufficiency

Standard MLLR and CMLLR estimate a linear transformation matrix to shift model parameters toward the target speaker's characteristics. When only one or two utterances are available, there is insufficient evidence to reliably estimate all coefficients in these matrices. Consequently, the model overfits the noise or specific nuances of the limited data, degrading performance on subsequent speech. While existing solutions like Maximum A Posteriori (MAP) adaptation or eigen-voices exist, they often require complex pre-computed priors or basis vectors.

Methodology: Ridge Adaptation

The authors introduce Ridge MLLR and Ridge CMLLR, which apply L2 regularization to the adaptation objective functions. By adding a penalty term based on the square of the transformation coefficients (the Frobenius norm), the optimization process is constrained to keep these coefficients small.

This "shrinkage" effect prevents the transformation matrix values from swinging wildly to fit a single noisy utterance. Instead, the parameters are "shrunk" toward zero (or the identity transform), ensuring that the model does not change too drastically without sufficient evidence. The authors derived closed-form solutions for these ridge-regularized versions, maintaining the computational efficiency required for real-time applications.

Distribution of transformation coefficients in Ridge MLLR, showing the shrinkage effect toward zero (more negative log values) compared to standard MLLR.

Experimental Evaluation

The proposed methods were evaluated on the 5k-WSJ0 (Wall Street Journal) task. The system was trained on the SI-84 set and tested on the Nov92 evaluation set. The researchers focused on a challenging "utterance-by-utterance" unsupervised adaptation scenario, where the system must adapt to a speaker using only the single sentence it just heard, without ground-truth transcripts.

Key Results and Insights

The experimental results demonstrate that ridge regularization significantly improves adaptation performance over standard methods. The most effective configurations used a regularization parameter ($\lambda$) between 100 and 500.

Word Error Rate (WER) results for Ridge CMLLR across various regularization strengths ($\lambda$).

As shown in the table above, Ridge CMLLR achieved a Word Error Rate (WER) of 4.46% at $\lambda=200$, representing a 12.2% relative reduction compared to the non-adapted baseline and a significant improvement over standard CMLLR (4.78%). Similarly, Ridge MLLR provided a 7.1% relative reduction in WER over standard MLLR.

A key insight from the study is that ridge regularization is particularly effective when applied to the rotation component of the transformation while allowing the bias (shift) more freedom. This prevents nonsensical rotations of the acoustic space when data is sparse.

Conclusion

By borrowing shrinkage techniques from classical statistics, the authors developed a simple yet powerful method for robust ASR adaptation. Ridge MLLR and Ridge CMLLR provide a "prior-free" way to stabilize model updates, making them ideal for real-world applications like voice search where only minimal user data is available. Future work may explore L1 regularization (LASSO) for sparse transformations or extending these techniques to covariance matrix estimation.

Summary updated

Abstract

We propose a parameter shrinkage adaptation framework to estimate models with only a limited set of adaptation data to improve accuracy for automatic speech recognition, by regularizing an objective function with a sum of parameter-wise power q constraint. For the first attempt, we formulate ridge maximum likelihood linear regression (MLLR) and ridge constraint MLLR (CMLLR) with an element-wise square sum constraint to regularize the objective functions of the conventional MLLR and CMLLR, respectively. Tested on the 5k-WSJ0 task, the proposed ridge MLLR and ridge CMLLR algorithms give significant word error rate reduction from the errors obtained with standard MLLR and CMLLR in an utterance-by-utterance unsupervised adaptation scenario.
BibTeX
@inproceedings{li2010shrinkage,
  author = {Jinyu Li and Yu Tsao and Chin-Hui Lee},
  title = {{Shrinkage model adaptation in automatic speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-478},
}
Figure