Learning new word pronunciations from spoken examples

Ibrahim Badr, Ian McGraw, James R. GlassInterspeech 2010 · DOI

Introduction

Modern Automatic Speech Recognition (ASR) systems rely heavily on data-driven acoustic and language models. However, the lexicon—the mapping between a word's orthographic form (graphemes) and its phonetic realization (phonemes)—remains a significant bottleneck. Traditionally, these lexicons are manually curated by experts, a process that is costly, slow, and difficult to scale for specialized domains or evolving vocabularies. While Grapheme-to-Phoneme (G2P) models offer an automated alternative, they frequently produce errors. This paper proposes a Bayesian framework to refine these predictions using a small number of spoken examples, effectively bridging the gap between automated predictions and expert-level accuracy.

Methodology

The researchers introduce the Pronunciation Mixture Model (PMM), a flexible approach that integrates acoustic evidence with a prior Graphone model (a joint sequence model of letters and sounds). Unlike traditional methods that might assume a single "correct" pronunciation, the PMM uses the Expectation-Maximization (EM) algorithm to learn weights for multiple possible pronunciations. This allows the system to capture valid variations in speech (e.g., "either" starting with "ee" or "eye").

The study also evaluates a Cascading Recognizers model, which treats the integration of acoustic evidence as a sequential process. While effective, this model is more susceptible to noise, as a single outlier utterance can negatively impact the entire chain of recognition.

Experimental Setup

The models were tested across two distinct datasets:

  1. PhoneBook Corpus: A clean dataset of isolated words recorded over telephone lines.
  2. Amazon Mechanical Turk (AMT) Corpus: A noisier, crowdsourced dataset designed to test the robustness of the models in real-world conditions.

An internal dictionary of 150,000 entries served as the training data for the initial G2P (graphone) prior.

Results and Discussion

The experimental results demonstrate that both the Cascading and PMM models can surpass expert-level performance with remarkably little data.

WER performance of Cascaded Recognizers and PMM compared to baselines.

As shown in the figure above, the G2P-only baseline starts at a 16.7% Word Error Rate (WER). Both proposed models reach the "expert" baseline of 12.4% WER with as few as three spoken examples. Notably, the PMM approach proved more robust to the noise present in the AMT dataset. By allowing for multiple baseforms (an average of 2.1 per word), the PMM achieved a WER of 11.0%, significantly outperforming the manually-crafted expert lexicon.

Qualitative analysis reveals how the PMM corrects systematic errors in standard dictionaries or G2P models.

Comparison of dictionary baseforms versus top PMM-generated baseforms.

The table above illustrates specific instances where the PMM identified more accurate or common phonetic realizations than the expert dictionary, such as correcting vowel sounds or consonant voicing in names and specialized terms.

Conclusion

This research highlights the viability of using small sets of spoken examples—even noisy, crowdsourced ones—to automatically generate high-quality ASR lexicons. The Pronunciation Mixture Model, in particular, offers a robust and scalable solution for maintaining up-to-date dictionaries without the need for constant expert intervention. Future work may explore joint training of lexicons and acoustic models to further optimize recognition performance in continuous speech environments.

Summary updated

BibTeX
@inproceedings{badr2010learning,
  author = {Ibrahim Badr and Ian McGraw and James R. Glass},
  title = {{Learning new word pronunciations from spoken examples}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-629},
}
Figure