Rapid Nonlinear Speaker Adaptation for Large-Vocabulary Continuous Speech Recognition

Zoi Roupakia, Anton Ragni, Mark J. F. GalesInterspeech 2012 · DOI

Introduction

Speaker adaptation is a critical component of modern Automatic Speech Recognition (ASR) systems, allowing models to adjust to the unique acoustic characteristics of individual users. While linear transformation techniques like Constrained Maximum Likelihood Linear Regression (CMLLR) are standard, they often fail to capture the inherent nonlinearities of human speech. Kernel Eigenvoice Adaptation (KEA) was introduced to address this limitation, but its practical application has been hindered by extreme computational demands. This paper presents a probabilistic framework to create compact nonlinear models that retain the accuracy of KEA while significantly reducing decoding overhead.

The Computational Challenge of KEA

The primary drawback of KEA is the "expansion" of the model. In a typical setup, KEA can increase the number of Gaussian components per state by orders of magnitude. For instance, a system might jump from 12 components per state to 384, making it 32 times more computationally expensive to evaluate likelihoods during decoding. This "curse of kernel methods" makes real-time deployment in large-vocabulary continuous speech recognition (LVCSR) nearly impossible without optimization.

Probabilistic Model Compression

To bridge the gap between high-performance expanded models and efficient baseline models, the authors propose a framework for compact nonlinear adaptation. The core of this approach is a clustering algorithm based on Kullback-Leibler (KL) divergence.

Instead of using heuristic pruning, the system groups the expanded set of Gaussians into a smaller, "target" set of components. By minimizing the KL divergence between the original expanded GMM and the new compact GMM, the framework ensures that the probabilistic integrity of the model is preserved. Since KL divergence for GMMs lacks a closed-form solution, the authors utilize the Matched Pair Bound Approximation (MPBA) to facilitate efficient optimization.

Figure 3: Visualization of the clustering process, where expanded Gaussian components are grouped into compact sets (A, B, C, D) to reduce model complexity.

Parameter Refinement and pSEMIT

Clustering alone is insufficient to maintain high accuracy. The authors demonstrate that "refining" the parameters—updating the weights, means, and covariances of the new clusters—is essential to recover information lost during compression.

However, refinement often leads to full-covariance matrices, which are slower to process than the diagonal-covariance matrices used in standard ASR. To solve this, the authors employ Predictive Semi-Tied Transforms (pSEMIT). This technique diagonalizes the refined matrices, restoring the decoding speed to baseline levels while attempting to preserve the benefits of the refined nonlinear adaptation.

Experimental Results

The proposed methods were evaluated on a 76-hour Large-Vocabulary Conversational Telephone Speech (CTS) task. The baseline Speaker Independent (SI) system achieved a Word Error Rate (WER) of 34.6%. The full, computationally expensive KEA system improved this to 33.3%.

The compact KEA system (using 48 components) achieved a WER of 33.6%, successfully retaining the majority of KEA's gains while being 8 times faster than the full expanded model. When pushed to the baseline size of 12 components, the system still outperformed the SI baseline, demonstrating the effectiveness of the probabilistic clustering approach.

Figure 6: Word Error Rate (WER) as a function of the number of regression classes used in the adaptation process.

As shown in the table above, increasing the granularity of adaptation classes (up to state-level) provides incremental improvements in WER, though the gains from pSEMIT were more modest than initially hypothesized.

Conclusion

This research demonstrates that nonlinear speaker adaptation can be made practical for large-scale ASR tasks. By moving from heuristic pruning to a probabilistic KL-divergence-based clustering framework, the authors achieved a 32-fold reduction in decoding cost. While a small performance gap remains between compact and full KEA models, the ability to deploy nonlinear adaptation with the efficiency of a standard diagonal-covariance system represents a significant step forward for real-time speech recognition.

Summary updated

BibTeX
@inproceedings{roupakia2012rapid,
  author = {Zoi Roupakia and Anton Ragni and Mark J. F. Gales},
  title = {{Rapid Nonlinear Speaker Adaptation for Large-Vocabulary Continuous Speech Recognition}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-486},
}
Figure