Adaptation in the pronunciation space for non-native speech recognition

Georg Stemmer, Stefan Steidl, Christian Hacker, Elmar NöthInterspeech 2004

TL;DR

This paper introduces a novel HMM interpolation technique to improve automatic speech recognition (ASR) for non-native speakers. By interpolating acoustic models of native speakers, they can better approximate the pronunciation variations found in non-native speech, leading to significant reductions in word error rates. This method offers an effective way to adapt ASR systems without requiring large amounts of non-native training data.

What Problem Does This Paper Address?

Automatic speech recognition systems often struggle with non-native accents. Studies have shown that word error rates (WER) can more than double when a system trained on native speakers is used for non-native speech. This is because non-native pronunciations can deviate significantly from native ones, and these deviations are highly variable depending on the speaker's native language and proficiency. Traditional approaches, like collecting more data or using more complex acoustic models, are often impractical due to the sheer diversity of non-native accents and the difficulty in obtaining sufficient data for each specific case. This paper aims to bridge this gap by developing an adaptation method that leverages existing native speech models to better handle non-native speech.

Methods

The core of the proposed method is HMM interpolation. The authors hypothesize that for any given non-native pronunciation of a speech sound, there's at least one native pronunciation that is acoustically similar. They leverage this by interpolating the parameters (specifically, mixture weights in semi-continuous HMMs) of native acoustic models to create new models that better represent the non-native speaker's pronunciation.

The process involves:

  1. Identifying potential interpolation partners: For each native HMM state, they consider a set of other native HMM states as potential partners.
  2. Estimating interpolation weights: Using a small amount of training data (either for a group of speakers or a single speaker in an unsupervised manner), they estimate optimal weights for interpolating the parameters of the native HMM states. This is framed as a learning problem, often solved using the Expectation-Maximization (EM) algorithm.
  3. Applying interpolated models: The resulting interpolated HMMs are then used in the ASR system.

This approach is distinct from methods that directly substitute HMM states or adapt output densities (like MLLR or MAP), as it smooths the pronunciation space by blending existing native models.

Datasets

  • Native Speech Corpora:
    • Verbmobil German Corpus: Used for training baseline German ASR systems (27.7 hours of speech from 578 speakers).
  • Verbmobil English Corpus: Used for training baseline English ASR systems (22 hours of speech from 260 speakers).
  • Non-Native Corpora:
    • BAS Strange Corpus 1 ("Accents"): Used for non-native read German speech. It contains 1.25 hours of speech from 72 non-native speakers (from 50 countries) and 16 native German speakers for reference. The text used was "The Northwind and the Sun."
    • Verbmobil Denglish Corpus: Used for non-native spontaneous English speech. It comprises 1.5 hours of speech from 44 German speakers. This corpus was split into disjoint training and test sets (22 speakers each).

Key Insights and Ideas

  • Native models are sufficient for adaptation: The fundamental assumption that native acoustic models can be adapted to approximate non-native pronunciations is validated by the experimental results.
  • Interpolation smooths the pronunciation space: Instead of rigid substitutions, blending native models allows for a more nuanced representation of non-native sounds that fall "between" native phonemes.
  • Data-driven partner selection is crucial: The method automatically identifies which native models are most relevant for adapting to a specific non-native accent, making it adaptable to diverse linguistic backgrounds.
  • Unsupervised adaptation is feasible: The technique can be applied to individual speakers without prior knowledge of their accent, which is essential for real-world dialogue systems.
  • Language model adaptation complements acoustic adaptation: Combining acoustic model interpolation with language model adaptation further boosts recognition performance, especially for spontaneous speech.

Key Results

The paper reports significant improvements in word error rates (WER) across different scenarios:

  • Non-native German (BAS Corpus):
    • Baseline WER: 34.0%
    • Interpolated (1 partner): 28.3% (16.8% relative improvement)
    • Interpolated (50 partners): 26.4% (22.4% relative improvement)
  • Non-native English (Verbmobil Denglish Corpus):
    • Baseline WER: 65.6%
    • Interpolated (1 partner): 61.6% (6.1% relative improvement)
    • Interpolated + Language Model Adaptation: 57.2% (12.8% relative improvement)

These improvements were found to be statistically significant. The results also showed that the interpolation method was particularly effective for speakers with higher baseline error rates.

Limitations and Future Work

  • Limitations:
    • The paper notes that in some cases, especially for speakers with already low WER, the interpolated system might perform slightly worse than the baseline. This is attributed to the baseline system potentially using polyphones while the interpolated system might rely on monophones. The authors suggest using a confidence measure to apply adaptation only when beneficial.
  • While the method adapts acoustic models, it doesn't alter the Gaussian densities themselves. The authors suggest this could be a limitation compared to methods like MLLR.
  • Future Work:
    • Combining with other adaptation techniques: The authors propose investigating how their HMM interpolation method can be combined with established speaker adaptation techniques like Maximum A Posteriori (MAP) and Maximum Likelihood Linear Regression (MLLR). They believe this could lead to further improvements, as MLLR adapts densities while interpolation adapts model structures.
  • Further exploration of language model adaptation: While language model adaptation was used and showed benefits, the paper defers detailed discussion to other works. Future work could delve deeper into optimizing this aspect.

Summary updated

BibTeX
@inproceedings{stemmer2004adaptation,
  author = {Georg Stemmer and Stefan Steidl and Christian Hacker and Elmar Nöth},
  title = {{Adaptation in the pronunciation space for non-native speech recognition}},
  booktitle = {Interspeech},
  year = {2004},
}
Figure