Abstract
Lucas F.E. Ashby, Travis M. Bartley, Simon Clematide, Luca Del Signore, Cameron Gibson, Kyle Gibson, Yeonju Lee-Sikka, Peter Makarov, Aidan Malanoski, Sean Miller, Omar Ortiz, Reuben Raff, Arundhati Sengupta, Bora Seo, Yulia Spektor, Winnie Yan. Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology.
2021.
Introduction
Grapheme-to-Phoneme (G2P) conversion—the process of mapping written text to its spoken equivalent—is a foundational component of modern speech technology, including text-to-speech (TTS) and automatic speech recognition (ASR). While G2P research has historically focused on high-resource languages like English, the second SIGMORPHON shared task expands this scope to 21 diverse languages. The task aims to address two primary challenges: the scarcity of high-quality training data for most of the world's languages and the inherent noise in community-sourced datasets like Wiktionary.
Methodology
The shared task organizers provided a robust baseline model based on a neural transducer system using an imitation learning paradigm. Unlike standard sequence-to-sequence models, this transducer learns explicit edit actions (insert, delete, substitute), which better aligns with the monotonic nature of G2P mapping.
Participating teams introduced several innovative strategies:
- Ensembling: The top-performing English system (Dialpad) utilized a massive ensemble of seven distinct models.
- Data Augmentation: The AZ team generated synthetic training examples by extracting substrings from existing words.
- Linguistic Heuristics: The UBC team incorporated explicit syllable boundaries and language-specific penalties for errors in diacritics or vowel length.
- Error Analysis via Covering Grammars: A significant contribution of this work is the use of finite-state "covering grammars" to distinguish between model failures and legitimate orthographic ambiguities (e.g., the dual pronunciation of 'e' in Italian).
Datasets and Resource Tiers
The task utilized WikiPron, a multilingual database extracted from Wiktionary. To evaluate model robustness across different data availability scenarios, the task was divided into three tiers:
- High-resource: ~41,000 words (Mainstream American English).
- Medium-resource: 10,000 words per language (10 languages).
- Low-resource: 1,000 words per language (10 languages).
The dataset featured a wide array of scripts, including Latin, Cyrillic, Greek, Georgian, Armenian, Khmer, and Japanese (Hiragana).
Key Results and Insights
The results across the 21 languages revealed several critical insights into the state of multilingual G2P.
The Impact of Data Quality
A notable success was seen in Georgian. Previously considered a difficult language in the 2020 task, improved data cleaning and quality assurance led to multiple systems achieving a 0.0% Word Error Rate (WER) in 2021. This underscores that model performance is often a direct reflection of data cleanliness.
Baseline Performance
In the medium-resource category, the organizers' imitation-learning baseline proved remarkably difficult to beat. None of the submitted systems consistently outperformed it, suggesting that for mid-sized datasets, a well-tuned, task-specific architecture may be superior to generic large-scale neural approaches.
Performance Summary
The high-resource English task remains challenging due to irregular orthography; the top system achieved a 37.43% WER, an 11% relative improvement over the baseline. In the low-resource tier, the best system (UBC-2) achieved a 4% relative reduction in WER.
Discussion and Future Directions
The shared task highlighted a persistent "performance gap" in low-resource settings. Current neural architectures struggle to reach peak performance when training data is limited to approximately 800 examples (the training split of the 1,000-word set).
Furthermore, while the task allowed for the inclusion of morphological data (e.g., lemmas or part-of-speech tags), no participating team successfully integrated this information. The authors suggest that leveraging such "extra-linguistic" context—knowing whether a word is a noun or a verb—could be essential for resolving ambiguities in languages like French or English where pronunciation is tied to grammatical function.
Summary updated