HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories

Akhilesh Kakolu Ramarao, Yulia Zinova, Kevin Tang, Ruben van de VijverNAACL 2022

TL;DR

This paper presents the HeiMorph team's submission to a SIGMORPHON shared task focused on how machines learn morphological inflection, particularly in low-resource scenarios. They explored data hallucination techniques with a Transformer architecture, finding that a phonotactically-aware "bigram" hallucination method improved performance, especially when applied judiciously.

What Problem Does This Paper Address?

Morphological inflection, the process of changing a word's form to convey grammatical information (like tense or number), is a fundamental aspect of language. For machines, learning these patterns is challenging, especially when training data is scarce, as is common for many languages. This paper tackles the problem of morphological acquisition trajectories in low-resource settings, specifically investigating how models learn inflectional patterns and whether data augmentation techniques can effectively bridge the gap caused by limited training examples. The authors are particularly interested in whether a "U-shaped" learning curve, sometimes observed in child language acquisition (where initial correct forms are followed by overgeneralization and then mastery), is evident in machine learning models.

Methods

The HeiMorph team employed a standard Transformer architecture for their models, utilizing the Fairseq toolkit. Key components include a 4-layer encoder-decoder with 4 attention heads, a 256-dimensional embedding size, and a 1024-dimensional hidden layer size. They trained these models using the Adam optimizer with specific hyperparameters like label smoothing and gradient clipping.

The core novelty lies in their exploration of data hallucination techniques. While the shared task provided a baseline using a vanilla Transformer with data augmentation, HeiMorph focused on a specific type of hallucination. They implemented a "bigram hallucination" method, which differs from previous approaches by ensuring that randomly replaced characters within a word segment are phonotactically attested in the original training data. This means the generated forms are more likely to sound like plausible words in the language, unlike more random character substitutions. They compared this with a "unigram" hallucination (random character replacement) and a baseline without hallucination.

Datasets

The experiments were conducted using data provided by the SIGMORPHON 2022 shared task (Part 2). This dataset covers two inflectional patterns: English past tense, German noun plurals, and Arabic noun plurals. The data is characterized by extreme sparsity, with training set sizes ranging from as few as 100 to 1,000 samples across three languages: English, German, and Arabic.

Key Insights and Ideas

  • No U-shaped Development: Contrary to some observations in child language acquisition, the authors found no evidence of a U-shaped learning trajectory in their machine models. This suggests that machine learning models might learn differently than human children, at least in this specific task.
  • Bigram Hallucination is Beneficial, but Nuanced: The phonotactically-aware "bigram" hallucination method generally outperformed the "unigram" (random) hallucination and the baseline. This highlights the importance of generating linguistically plausible synthetic data.
  • Quantity Matters (but not always more): For English and Arabic, generating fewer hallucinations (1,000) was often better than generating more (10,000), especially when combined with larger training set sizes. This indicates that excessive or poorly generated synthetic data can be detrimental.
  • Language-Specific Performance: The effectiveness of hallucination varied across languages. While bigram hallucination showed promise for English and Arabic, its impact on German was less pronounced or dependent on specific training conditions.

Key Results

The HeiMorph models achieved competitive results on the SIGMORPHON 2022 shared task. Their best models yielded average test accuracies of 55.5% on Arabic, 67% on German, and 73.38% on English. Compared to the neural baseline, their models showed improvements of 2.66% on Arabic, 8.69% on German, and 4.5% on English. Tables 1 and 2 provide detailed accuracy and Levenshtein distance metrics on the development and test sets, respectively, for various training set sizes and hallucination strategies.

Limitations and Future Work

The authors acknowledge that their development decisions were made with an incomplete picture of all experimental conditions. They also note that the effectiveness of hallucination is highly dependent on the specific language and inflectional pattern, suggesting it's not a one-size-fits-all solution. Future work could involve more in-depth analysis of the error patterns to further refine hallucination strategies and explore their application to a wider range of morphological phenomena and languages. The paper also implicitly suggests further investigation into the differences between machine and human language acquisition trajectories.

Summary updated

BibTeX
@inproceedings{ramarao2022heimorph,
  author = {Akhilesh Kakolu Ramarao and Yulia Zinova and Kevin Tang and Ruben van de Vijver},
  title = {{HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure