Speaker adaptation based on two-step active learning

Koichi Shinoda, Hiroko Murakami, Sadaoki FuruiInterspeech 2009

TL;DR

This paper introduces a novel two-step active learning strategy for speaker adaptation in speech recognition. By intelligently selecting adaptation sentences that target a speaker's specific recognition error patterns, the method significantly improves adaptation efficiency compared to random sentence selection, requiring fewer utterances for comparable performance gains.

What Problem Does This Paper Address?

Speaker adaptation aims to improve speech recognition accuracy for individual users by fine-tuning a general acoustic model with a small amount of their speech data. The challenge lies in maximizing the benefit of this adaptation data, especially when the amount of available data is limited. Traditional supervised adaptation often relies on pre-defined sentence sets, which may not effectively address a specific speaker's unique acoustic characteristics or recognition error tendencies. This can lead to suboptimal adaptation, requiring more data than necessary. The authors tackle the problem of efficiently selecting the right sentences for adaptation to achieve better performance with less data.

Methods

The proposed method employs a two-step active learning approach for supervised speaker adaptation:

  1. Initial Adaptation and Phone-Error Distribution Estimation: A small set of initial adaptation sentences is collected from the target speaker. A speaker-independent (SI) model is used to perform phone recognition on these utterances. From these results, a "phone-error distribution" is derived, highlighting which phone classes the speaker is most prone to misrecognizing. This is calculated as $P(u_i) = r(u_i) / \sum_{j=1}^{n} r(u_j)$, where $r(u_i)$ is the phone-error rate for phone $u_i$.
  2. Intelligent Sentence Selection and Second Adaptation: A larger pool of candidate adaptation sentences is prepared beforehand. For each sentence in this pool, a "phone distribution" is calculated, representing the frequency of each phone class within that sentence. The core of the active learning strategy is to select sentences whose phone distributions are "closest" to the speaker's estimated phone-error distribution. This closeness is measured using Kullback-Leibler Divergence (KLD). Sentences with the smallest KLD values are chosen as additional adaptation data.

Finally, the speaker's utterances for both the initial and the selected additional sentences are used to adapt the acoustic model, typically using techniques like Maximum Likelihood Linear Regression (MLLR). The paper focuses on a "batch adaptation" approach where all selected sentences are used at once.

Datasets

  • Japanese Read-Speech Database (JNAS): This database consists of Japanese newspaper article sentences and phonetically balanced sentences spoken by adults and seniors.
  • Training: 522 speakers (261 per gender) were used for training the initial acoustic models.
  • Testing: 44 speakers (22 per gender) were used for testing.
  • Adaptation Data Design: For each test speaker, 100 sentences were used: 60 for adaptation and 40 for testing. From the 60 adaptation sentences, 5 were used for the initial adaptation step, and the remaining 55 formed the sentence pool for selection in the second step.

Key Insights and Ideas

  • Speaker-Specific Error Patterns are Key: The fundamental insight is that speakers have individual acoustic characteristics that lead to predictable recognition errors. By identifying these patterns (the phone-error distribution), we can intelligently select adaptation data that directly addresses these weaknesses.
  • Kullback-Leibler Divergence as a Similarity Metric: KLD proves to be an effective measure for quantifying the similarity between a sentence's phonetic composition and the speaker's error profile, guiding the selection of the most beneficial sentences.
  • Active Learning for Data Efficiency: The method demonstrates that active learning, by selecting informative data points (sentences), can significantly reduce the amount of adaptation data needed to achieve a desired level of performance improvement.
  • Robustness to Initial Data: The proposed method's performance is shown to be relatively insensitive to the specific choice of the initial small set of adaptation sentences, suggesting its practical applicability.

Key Results

  • The proposed active learning method achieved a 1.1 absolute percentage point improvement in phone accuracy compared to random sentence selection when using 15 additional adaptation sentences.
  • The accuracy improvement from random selection was largest when 15 sentences were used.
  • The proposed method generally outperformed random selection across various numbers of adaptation utterances, with accuracies converging as the number of utterances increased.
  • The average phone accuracy for the speaker-independent model was 64.0%. The proposed method achieved accuracies in the low to mid-70s for the tested number of utterances.

Limitations and Future Work

  • Batch Adaptation: The evaluation used batch adaptation, which is simpler to implement but may not be as efficient as sequential adaptation. The authors suggest sequential adaptation as a future direction.
  • Limited Adaptation Pool Size: The study used a relatively small adaptation sentence pool (55 sentences). The authors believe that providing more choices in the sentence selection process could lead to further improvements.
  • Future Plans:
    • Apply the method to large-vocabulary continuous speech recognition using triphone HMMs.
  • Investigate sequential adaptation for potentially greater efficiency.
  • Develop an online evaluation scheme with a large text-only database and dynamic sentence determination based on previous recognition results.

Summary updated

BibTeX
@inproceedings{shinoda2009speaker,
  author = {Koichi Shinoda and Hiroko Murakami and Sadaoki Furui},
  title = {{Speaker adaptation based on two-step active learning}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure