TL;DR
This paper introduces the SIGMORPHON 2020 Shared Task, a multilingual grapheme-to-phoneme (G2P) conversion challenge that aimed to push the boundaries of G2P systems across 15 languages. Despite strong neural baselines, top-performing systems achieved only an 18% relative reduction in word error rate, highlighting the persistent difficulty of G2P, especially for low-resource languages.
What Problem Does This Paper Address?
Accurate pronunciation is crucial for many speech technologies, like automatic speech recognition and text-to-speech synthesis. Grapheme-to-phoneme (G2P) conversion, the process of mapping written characters to their corresponding sounds, is a fundamental component of these systems. While rule-based approaches were once common, modern systems increasingly rely on machine learning. However, most research has focused on well-resourced languages like English, leaving a significant gap in our understanding and capabilities for the vast majority of the world's languages, which have diverse writing systems and limited available pronunciation data. This paper addresses this gap by organizing a shared task and releasing a new multilingual dataset to encourage research on G2P for a wider range of languages.
Methods
The SIGMORPHON 2020 Shared Task focused on multilingual G2P conversion. Participants were tasked with building systems that take a sequence of graphemes (written characters) as input and output a sequence of phonemes (sounds) for one of fifteen languages. The task leveraged data from WikiPron, a large multilingual resource of grapheme-phoneme pairs extracted from Wiktionary. To ensure a fair comparison and focus on the core G2P task, words with multiple pronunciations were excluded. The data for each language was downsampled to 4,500 examples, creating a "medium-resource" setting.
The paper also describes three baseline systems:
- Pair n-gram model: A finite-state approximation of a hidden Markov model, serving as a traditional baseline.
- Encoder-decoder LSTM: A neural network sequence-to-sequence model with a bidirectional LSTM encoder and a unidirectional LSTM decoder, utilizing an attention mechanism.
- Encoder-decoder Transformer: Another neural sequence-to-sequence model, replacing recurrence with multi-head self-attention layers.
These baselines were made available to participants to ensure reproducibility and provide a benchmark.
Datasets
The primary dataset for the shared task was derived from WikiPron (Lee et al. 2020), a massively multilingual resource of grapheme-phoneme pairs extracted from Wiktionary. The task included data for fifteen languages and scripts, chosen to represent a wide variety of script types. For evaluation, the data was split into training, development, and testing sets. A key aspect of the dataset preparation was downsampling to 4,500 examples per language to create a medium-resource setting. Additionally, the paper mentions the use of UniMorph 2 (Kirov et al. 2018) to ensure that inflectional variants of lemmas were kept within the same data split.
Key Insights and Ideas
- G2P remains challenging, even with neural networks: Despite advancements in deep learning, the task of G2P conversion across multiple languages proved to be significantly harder than anticipated. The best systems only achieved a modest improvement over strong neural baselines, indicating that G2P is far from solved, especially in a multilingual setting.
- Data diversity is key, but not a silver bullet: The task deliberately included languages with diverse script types (alphabetic, syllabary, alphasyllabary) and origins. While this diversity is crucial for advancing G2P research, it also highlights how writing system characteristics can significantly impact conversion difficulty.
- Preprocessing matters: Several teams found success by employing preprocessing steps, such as romanizing Korean Hangul characters or decomposing them into constituent jamo. This suggests that adapting the input representation to be more amenable to G2P models can yield substantial gains.
- The limitations of WikiPron data: The paper acknowledges that inconsistencies or errors in the WikiPron data itself may have contributed to the observed error rates, suggesting a need for improved quality assurance for such resources.
- No one-size-fits-all solution: The results indicate that G2P technologies are not yet language-agnostic, and different approaches may be more suitable for different languages.
Key Results
- Nine teams submitted a total of 23 systems.
- The best systems achieved, on average, an 18% relative reduction in word error rate (WER) compared to the strong neural sequence-to-sequence baselines (LSTM and Transformer).
- The LSTM baseline generally outperformed the Transformer baseline in this medium-resource setting.
- For some languages, the best baseline system did not achieve the best phone error rate (PER), though the overall ranking of best systems remained consistent.
- The IMS team achieved the lowest average WER and PER across all submissions.
- Even on the "easiest" language, the best submission had a WER of 0.89%, and for three languages, no submission achieved an error rate below 20%.
- Korean, with its complex Hangul script and morphophonemic orthography, presented a particularly challenging case, with the best submission achieving a 45% relative WER reduction over its baseline.
Limitations and Future Work
- Data Sparsity: The task was designed as a "medium-resource" setting (4,500 examples per language). The authors acknowledge that more data would likely favor transformer models and potentially lead to better performance.
- WikiPron Data Quality: Inconsistencies and potential errors in the WikiPron data were reported by participants, which could inflate error rates and affect model training. The authors are developing quality assurance software for WikiPron and plan to apply large-scale consistency-enforcing edits.
- Language Agnosticism: The results suggest that current G2P technologies are not fully language-agnostic, and further research is needed to develop more universally applicable models.
- Future Work: The authors suggest exploring low-resource settings for G2P, investigating the use of task-specific resources like phonemic inventories and rule-based G2P systems, and incorporating morphological analyzers or lexicons. They also plan to apply consistency-enforcing edits to Wiktionary itself. The public release of all system outputs is intended to facilitate error analysis and future research.
Summary updated