TL;DR
This paper explores ensembling techniques for grapheme-to-phoneme (g2p) conversion, demonstrating that combining multiple models, even those with lower individual performance, can significantly improve accuracy. The authors highlight the challenges of noisy, dialectally diverse data in g2p tasks and propose a robust ensembling strategy that outperformed the baseline by a substantial margin.
What Problem Does This Paper Address?
Grapheme-to-phoneme (g2p) conversion, the process of translating written text into its spoken sound representation, is a fundamental component in speech science and technology. It's crucial for tasks like building pronunciation dictionaries for Automatic Speech Recognition (ASR) and speech synthesis systems. The SIGMORPHON 2021 Shared Task 1 focused on g2p, with a particular emphasis on a high-resource English-only sub-task. The challenge lies in the inherent noisiness and variability of real-world pronunciation data, often sourced from collaborative efforts like Wiktionary. This data can contain dialectal variations, inconsistent transcription broadness, and even outright errors, making it difficult for any single model to capture all nuances. The authors aim to tackle this by investigating how ensembling multiple g2p models can lead to more robust and accurate predictions, especially in the face of such data challenges.
Methods
The paper presents three main systems submitted by the Dialpad team:
- Dialpad-2 (Majority Vote Ensemble): This system is a simple majority-vote ensemble of three publicly available g2p systems: Phonetisaurus (WFST-based), Sequitur (joint-sequence model), and a neural sequence-to-sequence model from CMUSphinx. They retrained these models on the cleaned task data.
- Dialpad-3 (Strong Standalone Model): This is a Transformer-based neural sequence-to-sequence model (CMUSphinx g2p-seq2seq) that was retrained from scratch on the task's training data, as the pre-trained model used a different phoneme set (ARPABET vs. IPA).
- Dialpad-1 (Large Ensemble): This is the most complex system, combining two variations of the baseline model, the Dialpad-2 ensemble, and two additional seq2seq models (one biLSTM, one Transformer-based). This ensemble uses a word-level majority vote, with ties broken by selecting the prediction from the best standalone model on the development set.
Additionally, they explored:
- PAS2P (Pronunciation-Assisted Sub-words to Phonemes): This method uses pronunciation-assisted sub-word units as input to a Transformer-based seq2seq model, aiming to bias the mapping towards better grapheme-phoneme correspondences.
- Edit Distance Tie-Breaking: An alternative voting scheme for ensembling where ties are broken by selecting the prediction that minimizes edit distance to other tied predictions.
Datasets
The primary dataset used was from the SIGMORPHON 2021 Shared Task 1, specifically the high-resource English-only sub-task. This dataset consists of word/pronunciation pairs derived from the English portion of the WikiPron database. It contained 33,344 lines for training, and 4,168 lines each for development and testing. The authors note that this data, while extensive, exhibits significant noise and dialectal variation.
Key Insights and Ideas
- Ensembling is Powerful, Even with Sub-optimal Components: The most striking insight is that combining multiple models, even those that individually perform worse than the baseline, can lead to significant improvements. This suggests that different models capture different aspects of the g2p mapping, and their collective "wisdom" is greater than the sum of their parts.
- Data Noise is a Major Hurdle: The paper strongly emphasizes that the noisy and varied nature of the training data is a primary source of errors. Inconsistencies in transcription broadness, dialectal variations (like the cot/caught merger), and even outright misspellings pose significant challenges for g2p models.
- A Strong Standalone Model Can Be a Foundation: The CMUSphinx Transformer-based seq2seq model (Dialpad-3) proved to be a very strong standalone performer, even outperforming the Dialpad-2 ensemble. This highlights the effectiveness of modern neural architectures for g2p.
- Linguistically-Informed Preprocessing Helps: The PAS2P approach, which uses pronunciation-assisted sub-word units, shows promise by incorporating linguistic knowledge into the input representation.
Key Results
The Dialpad team's submissions achieved impressive results:
- Dialpad-3 (Standalone CMUSphinx Transformer): Achieved a test set WER of 41.58.
- Dialpad-2 (Majority Vote Ensemble): Achieved a test set WER of 41.72.
- Dialpad-1 (Large Ensemble): This system achieved the best result with a test set WER of 37.43. This represents a significant 10.75% absolute reduction in WER compared to the baseline, and approximately a 37% relative reduction.
- The edit-distance tie-breaking method, while not improving development set performance, consistently outperformed other ensembling methods on the test set.
Limitations and Future Work
- Data Noise: The authors acknowledge that the inherent noise and dialectal variations in the training data are a major limitation, leading to errors even when models agree.
- Oracle WER: The "oracle WER" (the theoretical best possible WER if the correct prediction was always available) was significantly lower (25.12 dev, 23.27 test) than the actual achieved WERs, indicating substantial room for improvement.
- Future Work: The authors suggest exploring:
- Optimal trade-offs between ensemble size and performance.
- More sophisticated ensembling and voting methods, potentially at the phone level on aligned outputs.
- Further investigation into the edit-distance tie-breaking scheme.
Summary updated