A comparative study of two kernel eigenspace-based speaker adaptation methods on large vocabulary continuous speech recognition

Roger Hsiao, Brian MakInterspeech 2005

TL;DR

This paper compares two kernel-based speaker adaptation methods, eKEV and KEMLLR, on a large vocabulary speech recognition task. Both methods significantly reduce word error rates compared to standard techniques, with KEMLLR offering slightly better accuracy but eKEV providing a much faster adaptation speed.

What Problem Does This Paper Address?

Speaker adaptation aims to quickly customize a general speech recognition model to a specific speaker using a small amount of their speech. Traditional methods like Maximum A Posteriori (MAP) and Maximum Likelihood Linear Regression (MLLR) are effective but struggle when adaptation data is extremely scarce (e.g., a few seconds). Eigenspace-based methods, like Eigenvoice (EV), address this by representing speakers in a lower-dimensional "eigenspace," reducing the number of parameters to adapt. However, these methods often assume linearity. This paper explores how to leverage kernel methods to capture potential nonlinearities in speaker characteristics, improving adaptation performance, especially for very limited adaptation data. The core challenge is to evaluate and compare these advanced kernelized methods on a challenging Large Vocabulary Continuous Speech Recognition (LVCSR) task, which was not previously demonstrated.

Methods

The paper focuses on two kernel eigenspace-based speaker adaptation methods:

  • Embedded Kernel Eigenvoice (eKEV): This method maps speaker supervectors (concatenated Gaussian means) into a high-dimensional feature space using a nonlinear kernel function. Kernel PCA is then applied in this feature space to derive "kernel eigenvoices." The adaptation involves finding a linear combination of these kernel eigenvoices that best represents the new speaker in the feature space. A key innovation is the "pre-imaging" step, which maps this representation back to the original input space to create an approximate speaker model. This pre-imaging step avoids direct computation in the high-dimensional space, making it more efficient.
  • Kernel Eigenspace-based MLLR (KEMLLR): Similar to eKEV, KEMLLR also uses kernel PCA in a high-dimensional feature space. However, instead of using Gaussian means, it vectorizes the MLLR transformation matrices learned for each speaker. Kernel PCA is then applied to these vectorized MLLR transforms. This method aims to capture nonlinear relationships within the MLLR transformations themselves.

Both methods utilize kernel PCA to derive eigenvoices in a kernel-induced feature space and then adapt speaker models. The primary difference lies in what constitutes the "speaker model" in the feature space: Gaussian means for eKEV and MLLR transforms for KEMLLR.

Datasets

  • Training: The standard SI-84 training set from the Wall Street Journal (WSJ0) corpus was used. This set comprises 83 speakers and approximately 14 hours of speech.
  • Evaluation: The nov'92 5K non-verbalized test set from WSJ0 was used for evaluation. This set contains 8 speakers, and adaptation data was randomly selected from their utterances to simulate 5s and 10s of adaptation speech.

Key Insights and Ideas

  • Kernel Methods Enhance Adaptation: Kernel methods can effectively capture nonlinearities in speaker characteristics, leading to improved adaptation performance, especially when adaptation data is scarce.
  • eKEV vs. KEMLLR Trade-offs: While KEMLLR generally achieves slightly higher accuracy, eKEV offers a significant speed advantage in adaptation due to its more efficient pre-imaging mechanism.
  • Porting to LVCSR is Feasible: The paper demonstrates that these kernelized methods, initially developed for smaller vocabulary tasks, can be successfully ported and applied to LVCSR tasks with triphones.
  • Handling Small Eigenvalues: A practical challenge in porting to LVCSR was the presence of small eigenvalues in the kernel matrix, which could cause numerical instability. The authors found that discarding these small eigenvalues resolved the issue.
  • Storage Efficiency: For LVCSR, using MLLR to create speaker-dependent models and then computing HMM means on-the-fly during adaptation is more storage-efficient than storing full SD HMMs.

Key Results

  • Significant Word Error Rate Reduction: Both eKEV and KEMLLR significantly outperformed standard MAP and MLLR adaptation on the WSJ0 5K-vocabulary task.
  • With 5s of adaptation data, eKEV and KEMLLR achieved word error rate reductions of 13.4% and 16.3%, respectively, compared to the speaker-independent model.
  • With 10s of adaptation data, KEMLLR achieved a 21.1% word error rate reduction, while eKEV achieved 14.11%.
  • KEMLLR's Accuracy Edge: KEMLLR consistently showed slightly better word accuracy than eKEV, especially with 10s of adaptation data.
  • eKEV's Speed Advantage: eKEV's adaptation speed was about 1/3 to 1/2 that of KEMLLR, making it more suitable for real-time applications where adaptation speed is critical. KEMLLR required significantly more eigenvoices (83 for WSJ0) compared to eKEV (10 for WSJ0).

Limitations and Future Work

  • Parameter Sensitivity: While the paper mentions that performance is not "very sensitive" to parameter settings once in the right order of magnitude, empirical tuning was still required.
  • KEMLLR's Eigenvoice Requirement: KEMLLR requires a much larger number of eigenvoices than eKEV, which can impact computational resources and adaptation speed.
  • Future Work: The paper implicitly suggests that further exploration of composite kernels and interpolation strategies might yield even better results. The trade-off between accuracy and speed between eKEV and KEMLLR also points to potential research in hybrid approaches or further optimization of each method.

Summary updated

BibTeX
@inproceedings{hsiao2005comparative,
  author = {Roger Hsiao and Brian Mak},
  title = {{A comparative study of two kernel eigenspace-based speaker adaptation methods on large vocabulary continuous speech recognition}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure