TL;DR
This paper presents the SIGMORPHON 2022 Shared Task, focusing on grapheme-to-phoneme (G2P) conversion in low-resource and cross-lingual settings. The task's novel subtask structure and expanded language pairs aim to shed light on data efficiency and the effectiveness of cross-lingual transfer for G2P modeling, revealing that while transfer learning helps, it doesn't fully bridge the gap with in-language data.
What Problem Does This Paper Address?
Accurately converting written text (graphemes) into their spoken sounds (phonemes) is a fundamental challenge in speech technology. While high-resource languages have ample data for training G2P models, many languages lack sufficient data, creating a "resource bottleneck." This paper addresses the need for effective G2P modeling in these low-resource scenarios. Specifically, it investigates how much information can be transferred from related, higher-resource languages (cross-lingual transfer) and how well models can learn G2P mappings intrinsically with limited data. The SIGMORPHON shared tasks have historically pushed the boundaries of this problem, and this iteration aims to provide clearer answers about data efficiency and the utility of cross-lingual transfer by standardizing datasets and test sets across different resource conditions.
Methods
The SIGMORPHON 2022 shared task featured three subtasks designed to isolate the effects of data availability and cross-lingual transfer:
- Cross-lingual Transfer: Participants received a small amount of data (100 words) in the target language and a larger amount (1000 words) in a related "transfer" language.
- Very-low Resource: Participants received only a small amount of data (100 words) in the target language, with no transfer language data.
- Low Resource: Participants received a larger amount of data (1000 words) in the target language, with no transfer language data.
All subtasks used the same 100-word test set for evaluation. The data preparation involved sampling from WikiPron, excluding words with fewer than two characters or phone segments, and removing words with multiple pronunciations. Data was then split into training, development, and test sets, with a constraint to ensure that inflectional variants of a lemma did not appear in different shards. Participants were also provided with morphological information from UniMorph, though no team utilized it.
Datasets
The primary dataset used for this task was WikiPron (Lee et al., 2020), a multilingual pronunciation database extracted from Wiktionary. For the shared task, 20 WikiPron languages were selected and organized into 10 target-transfer language pairs. Notably, two "surprise" pairs (Irish-Welsh and Burmese-Shan) were withheld until close to the deadline as a challenge. The data was curated to exclude English due to reported quality issues.
Key Insights and Ideas
- Cross-lingual transfer is beneficial, but not a silver bullet: While cross-lingual transfer significantly improved performance compared to the very-low resource setting, it did not fully close the gap with in-language examples. This suggests that while related languages offer valuable information, direct learning from the target language remains superior when sufficient data is available.
- Language difficulty varies greatly: The results highlighted substantial differences in how challenging G2P modeling is across languages, even for those with similar scripts or linguistic relationships. This underscores the complexity of G2P and the need for language-specific considerations.
- Non-neural approaches can still compete: In low-resource regimes, non-neural methods like HMMs with data augmentation showed competitive results, mirroring trends seen in other NLP tasks like machine translation. This suggests that for extremely data-scarce scenarios, simpler, well-tuned models can be effective.
- Gold data errors can impact results: The paper acknowledges that a portion of errors observed in top systems might stem from inaccuracies within the gold-standard pronunciation data itself, a known issue in G2P research.
Key Results
Three teams submitted a total of fifteen systems. The best systems achieved relative reductions in word error rate (WER) of up to 14% in the cross-lingual subtask and 14% in the very-low resource subtask compared to the baseline. The macro-averaged WER results across languages demonstrated the effectiveness of different approaches in each subtask. For instance, in the cross-lingual subtask, systems generally outperformed the baseline, with macro-averaged WERs around 75% compared to the baseline's 83.48%. In the very-low resource subtask, performance was more varied, with macro-averaged WERs around 74-79% for top systems versus the baseline's 85.20%. The low-resource subtask saw the most significant improvements, with macro-averaged WERs dropping to around 40-51% from a baseline of 36.88%.
Limitations and Future Work
The authors acknowledge that the data is a mix of broad and narrow transcriptions, which could contribute to variations in language difficulty. They suggest that further research is needed to quantify this impact. Future work could involve:
- Splitting test sets into frequent and infrequent words to better evaluate performance on different word types.
- Evaluating the downstream performance of G2P models in Automatic Speech Recognition (ASR) systems.
- Providing more explicit linguistic resources, such as information on phylogenetic/hegemonic relationships between languages, to aid model development.
- Investigating the role of typological information to improve modeling, especially for phenomena not observed in small sample sizes.
Summary updated