TL;DR
This paper tackles the challenge of accurately pronouncing foreign proper names in US English text-to-speech (TTS) systems. By incorporating knowledge of a name's likely language of origin into a statistical pronunciation model, they achieve a significant improvement in pronunciation accuracy, making synthesized speech sound more natural.
What Problem Does This Paper Address?
Text-to-speech (TTS) systems rely on dictionaries to convert written text into spoken words. However, these dictionaries are never complete, and a significant portion of "unknown" words in real-world text are proper names. These proper names often originate from diverse linguistic backgrounds, making their pronunciation in English a complex task. Standard pronunciation models struggle with these names, leading to unnatural or incorrect pronunciations. The authors hypothesize that understanding a proper name's linguistic origin can provide crucial clues for predicting its educated American English pronunciation, similar to how humans adapt their pronunciation based on perceived origin.
Methods
The core of the paper introduces a novel approach that augments a standard statistical pronunciation model (specifically, a CART decision tree) with information about a proper name's language of origin.
- Baseline Model: They start with a CART-based letter-to-sound (LTS) model trained on a large corpus of US English proper names (CMUDICT). This serves as their baseline.
- Language Identification: To determine a name's likely origin, they develop a language identifier. This identifier uses "Letter to Language Models" (LLMs), which are essentially trigram models trained on corpora from 26 different languages. For a given input name, the identifier outputs probabilities of it belonging to each of these languages.
- Feature Integration: Instead of building separate pronunciation models for each language (which would suffer from data sparsity), they integrate the language identification probabilities as features directly into the CART model. These features include the top predicted language, its probability, the second-best language, its probability, and the difference between these probabilities. This allows the CART model to dynamically use language origin information when it's most relevant for predicting pronunciation.
- Extended Phoneme Set: To better handle the nuances of foreign names, they also expand the set of allowed phoneme combinations beyond the standard CMUDICT set, incorporating variations and diphones observed in foreign name pronunciations.
Datasets
- Training/Testing Data: A lexicon of 56,000 proper names from Bell Labs' directory listings (approximately 20 years old), containing common US surnames and other names, sourced from CMUDICT. This dataset was split into 90% for training and 10% for testing.
- Language Corpora: 18 text corpora from the European Corpus Initiative Multilingual Corpus I (covering 13 languages) and 8 proper name corpora (Catalan, Chinese, Japanese, Korean, Polish, Thai, Tamil, and other Indian languages) were used to build the Letter to Language Models (LLMs).
Key Insights and Ideas
- "Educated Americanized" Pronunciation: The goal isn't to replicate the original foreign pronunciation but to model how an educated American English speaker would pronounce it. This is a crucial distinction for TTS systems.
- Language Origin as a Powerful Feature: The hypothesis that language origin significantly aids pronunciation prediction is strongly supported. This is analogous to human intuition, where knowing a name's likely origin guides pronunciation.
- Language Identifier for Feature Engineering: Using a probabilistic language identifier to generate features for a single, unified pronunciation model is more robust than attempting to build separate models for each language, especially given data sparsity.
- Potential for Outperforming Native Speakers: For certain names (e.g., Chinese), the system, armed with language origin knowledge, could potentially achieve higher accuracy than average native English speakers who lack this specific linguistic knowledge.
Key Results
- Quantitative Improvement: The model incorporating language origin features (PN-lang-5) achieved a 7.64% absolute improvement in word accuracy over the baseline CART model (PN-base-5), reaching 61.72% compared to 54.08%. This improvement was consistent across different CART tree stopping criteria.
- Qualitative Improvement (User Studies): In user studies, listeners showed a 17% preference for the model that incorporated language probability information (PN-lang-8) over the baseline (PN-base-8). This indicates that the improved accuracy translates to more natural-sounding speech.
Limitations and Future Work
- Data Fragmentation: The authors note that with a smaller stop value for the CART trees (e.g., 5), there might be data fragmentation, leading to less reliable estimates on the leaves. They addressed this by testing with a larger stop value (8), though the improvement was less pronounced.
- LLM Training Data: Ideally, LLMs should be trained on name-specific corpora rather than general text. However, their experiments showed that text-trained LLMs performed comparably or better than name-trained LLMs for some languages.
- Prior Probabilities: The authors suggest exploring the addition of "prior probabilities" for each language, which would represent a general likelihood of a name belonging to a language, independent of the name itself. This would be easier to estimate from news data than from name directories.
- Upper Bound of Accuracy: They acknowledge that there's still room for improvement and that the true upper bound of accuracy using this approach is yet to be determined.
- Human Consistency: They highlight that human language identification can be inconsistent (less than 50% confidence), suggesting that an automated system with language origin knowledge will likely be more consistent and perform better.
Summary updated