BME Submission for SIGMORPHON 2021 Shared Task 0. A Three Step Training Approach with Data Augmentation for Morphological Inflection

Gábor Szolnok, Botond Barta, Dorina Lakatos, Judit ÁcsACL 2021

TL;DR

This paper presents a novel three-step training approach combined with data augmentation techniques for morphological inflection, achieving the top submission in the SIGMORPHON 2021 Shared Task 0. Their method, while simpler than the Transformer baseline, demonstrates competitive performance and offers a practical solution for low-resource languages.

What Problem Does This Paper Address?

Morphological inflection, the process of generating different forms of a word based on grammatical context (e.g., "walk" -> "walks," "walked"), is crucial for natural language generation, especially in languages with rich morphology. The SIGMORPHON Shared Tasks focus on this challenge, particularly on the ability of models to generalize to unseen languages. This paper tackles the problem of building effective morphological inflection systems, especially for languages with limited training data, by proposing a structured training regime and innovative data augmentation strategies.

Methods

The core of the BME submission is a three-step training approach for a standard LSTM encoder-decoder model with attention.

  1. General Training: The model is initially trained on all available languages, incorporating a "copy" data augmentation technique. This step aims to build a general understanding of morphological patterns across diverse languages.
  2. Language Family Training: The model is then fine-tuned on each language family. Here, a "stem modification" augmentation technique is introduced, which involves replacing characters in the word stem with randomly sampled characters based on the language's unigram distribution. This helps the model learn language-specific nuances.
  3. Language Specific Training: Finally, the model is fine-tuned on individual languages. In this stage, a "template-based augmentation" method is employed, which uses identified morpheme patterns to generate new training examples.

The authors also explored a "template induction" method but found it did not improve overall results and thus excluded it from their final submission.

Datasets

The SIGMORPHON 2021 Task 0 Part 1 dataset comprises 38 languages from 12 language families. The languages vary significantly in size, with some having over 100,000 examples and others as few as 128. The paper highlights the challenge posed by "low-resource languages" (defined as having fewer than 1300 samples) and aims to mitigate this imbalance through data augmentation.

Key Insights and Ideas

  • Three-Step Training is Key: The hierarchical training approach (general -> family -> specific) significantly improves performance, allowing the model to leverage broad linguistic patterns before specializing.
  • Data Augmentation Matters (But Not Always): While "copy" augmentation proved highly beneficial, "stem modification" had mixed effects, sometimes even hurting performance on very low-resource languages like Ludic. This suggests that augmentation strategies need careful consideration and may require language-specific tuning.
  • Simplicity Can Be Competitive: The authors' LSTM-based model, despite being simpler than the Transformer baseline, achieved the highest submission score for the task, demonstrating that complex architectures aren't always necessary for strong performance.
  • Low-Resource Challenges Persist: Even with augmentation, the model's performance on low-resource languages can be significantly lower than on high-resource ones, indicating ongoing challenges in this area.

Key Results

The BME submission achieved an average accuracy of 81.51% with an average Levenshtein distance of 0.627. This was lower than the baseline (84.57% accuracy, 0.448 edit distance) but competitive given the model's simplicity. The paper reports that their submission outperformed the only other submission to the task by a large margin. Ablation studies showed that the "copy" augmentation technique was the most impactful, while "stem modification" and the training steps had more varied effects depending on the language.

Limitations and Future Work

The authors acknowledge that their LSTM model, while competitive, did not surpass the Transformer baseline. They also note that the "template-based augmentation" was computationally expensive and had mixed results. The paper suggests that further investigation into language-specific tuning of augmentation techniques and training steps would be beneficial for optimizing performance on individual languages. They also recommend trying these options individually for language-specific applications.

Summary updated

BibTeX
@inproceedings{szolnok2021bme,
  author = {Gábor Szolnok and Botond Barta and Dorina Lakatos and Judit Ács},
  title = {{BME Submission for SIGMORPHON 2021 Shared Task 0. A Three Step Training Approach with Data Augmentation for Morphological Inflection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure