Speaker adaptation based on nonlinear spectral transform for speech recognition

Toyohiro Hayashi, Yoshihiko Nankaku, Akinobu Lee, Keiichi TokudaInterspeech 2010

TL;DR

This paper introduces a novel speaker adaptation technique that uses a nonlinear spectral transform based on Gaussian Mixture Models (GMMs) to improve speech recognition accuracy. By combining GMM-based spectral transformations with Hidden Markov Models (HMMs) for recognition, the method achieves better performance than traditional linear methods like MLLR, especially in handling speaker variations.

What Problem Does This Paper Address?

Speech recognition systems often struggle when encountering speech from speakers different from those used during training. This mismatch, known as speaker variability, degrades performance. Speaker adaptation techniques aim to adjust the recognition model using a small amount of speech from the target speaker. Traditional methods like Maximum Likelihood Linear Regression (MLLR) use linear transformations, which can only capture a limited range of speaker-specific variations. While spectral transform techniques based on GMMs can model nonlinear transformations, they often don't effectively integrate with the HMMs used for recognition and can struggle to account for contextual dependencies in speech. This paper addresses the need for a speaker adaptation method that can model complex, nonlinear spectral transformations while also effectively incorporating contextual information and integrating seamlessly with the underlying HMM recognition framework.

Methods

The core of this paper is a novel speaker adaptation technique that combines HMMs for recognition with GMMs for spectral transformation. Instead of applying a single linear transform per state (as in MLLR), this method uses a GMM to model nonlinear spectral transformations. A key innovation is the definition of a new likelihood function that jointly considers the HMMs for recognition and the GMMs for spectral transformation. This allows for the estimation of nonlinear transforms that are adapted to the specific acoustic model states.

Furthermore, the authors introduce a mechanism to capture the dependency of these transforms on context. By incorporating state dependency into the mixture weights of the GMM, the transforms can vary based on the HMM state, similar to how MLLR uses regression classes. This allows the model to represent contextual information more effectively. The parameters of the GMM are estimated using an Expectation-Maximization (EM) algorithm. An approximation is proposed to manage the computational complexity arising from marginalizing out the transformed feature sequences.

Datasets

The experiments were conducted using the ATR Japanese speech database (B-set). For speaker-dependent experiments, one male speaker was used for training and another for adaptation. For speaker-independent experiments, four male and female speakers were used for training, with adaptation and evaluation performed on a subset of sentences from one of these speakers.

Key Insights and Ideas

  • Nonlinear Transforms for Better Adaptation: The paper demonstrates that nonlinear spectral transformations, modeled by GMMs, can capture speaker variations more effectively than linear transforms used in MLLR.
  • Integrated Framework: By defining a joint likelihood function, the proposed method seamlessly integrates spectral transformation with the HMM recognition process, allowing for more informed adaptation.
  • Contextual Dependency: The ability to model the dependency of transforms on context (via state-dependent GMM mixture weights) is crucial for improving adaptation performance, mirroring the benefits of MLLR's regression classes but within a nonlinear framework.
  • Soft Clustering: The EM-based estimation of GMM parameters allows for soft clustering of regression classes, offering a more nuanced approach to adaptation compared to hard assignments.

Key Results

  • Improved Phoneme Accuracy: The proposed nonlinear spectral transform (NLST) method consistently outperformed MLLR in speaker-dependent experiments, achieving higher phoneme recognition accuracy across different numbers of transform matrices (mixtures/classes). For example, in speaker-dependent tests, NLST showed gradual improvement with iterative updates, while MLLR plateaued.
  • Effectiveness in Speaker-Independent Settings: The method also proved effective in speaker-independent scenarios, outperforming MLLR.
  • Variance Transform Benefits: When comparing with CMLLR (which includes mean and variance transforms), the proposed method (CNLST) also showed superior performance, highlighting the benefit of its nonlinear approach to variance adaptation.

Limitations and Future Work

The authors acknowledge that the proposed method, while effective, involves significant computational cost due to the complexity of the likelihood function. They suggest that future work could focus on optimizing this computational aspect. Additionally, they propose exploring the estimation of variance bias parameters and further refining the clustering of weight parameters as avenues for future research.

Summary updated

BibTeX
@inproceedings{hayashi2010speaker,
  author = {Toyohiro Hayashi and Yoshihiko Nankaku and Akinobu Lee and Keiichi Tokuda},
  title = {{Speaker adaptation based on nonlinear spectral transform for speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure