AX Semantics’ Submission to the SIGMORPHON 2019 Shared Task

Andreas Madsack, Robert WeißgraeberACL 2019

TL;DR

This paper presents two neural network systems for morphological reinflection, a task that involves generating inflected word forms from a lemma and a set of grammatical features. The authors leverage the AllenNLP framework to build encoder-decoder models, focusing on reproducibility and ease of use. Their work highlights the challenges of low-resource languages and the effectiveness of transfer learning in this domain.

What Problem Does This Paper Address?

The SIGMORPHON 2019 Shared Task focuses on morphological reinflection, which is the process of generating the correct inflected form of a word given its base form (lemma) and specific grammatical information (like tense, number, gender, etc.). For example, given the lemma "sehen" (to see) and the features "past tense, 3rd person, plural," the system should output "sahen" (they saw).

A significant challenge in this task, especially compared to previous years, is the imbalanced data distribution. The shared task provides a large amount of data for "high-resource" languages and a very small amount for "low-resource" languages. The goal is to use the abundant data from high-resource languages to improve performance on the low-resource ones, which is a classic scenario for transfer learning. The authors aim to build robust, reproducible systems using a modern NLP framework (AllenNLP) to tackle this problem.

Methods

The authors developed two main systems, both based on an encoder-decoder architecture implemented using the AllenNLP framework.

  • System 1 (Softmax Baseline): This system is a straightforward implementation of a soft-attention encoder-decoder model, serving as a baseline. It uses AllenNLP's SimpleSeq2Seq component with dot-product attention. The input to the model is the lemma concatenated with a "next marker" and the morphological features as a string. The model is trained on the provided data for each language pair.
  • System 2 (Transfer Learning): This system builds upon System 1 but incorporates a transfer learning strategy. It first trains on all available data (both high-resource and low-resource languages) for a set number of epochs (10). After this initial phase, it switches to training only on the low-resource target language data. The switch is triggered by a lack of improvement in validation loss over a few epochs, indicating a point where further training on high-resource data might not be beneficial or could even be detrimental. This approach aims to leverage the general linguistic patterns learned from high-resource languages before fine-tuning on the specific target language.

Both systems operate at the character level, meaning they process words as sequences of characters rather than sub-word units or full words.

Datasets

The paper uses data provided for the SIGMORPHON 2019 Shared Task on Morphological Reinflection. This dataset consists of numerous language pairs, each with a high-resource language and a low-resource language. The task is to train on the high-resource data to improve performance on the low-resource language. The authors also mention training a version of System 1 with only the low-resource data to serve as a comparison.

Key Insights and Ideas

  • AllenNLP for Reproducibility: The authors emphasize their choice of AllenNLP for its built-in features for reproducibility and logging, which they found beneficial for their production systems.
  • Transfer Learning Effectiveness: System 2, employing a transfer learning approach, generally shows improved performance (lower loss) compared to System 1, especially in scenarios where the low-resource language benefits from the general linguistic knowledge learned from a high-resource counterpart.
  • Character-Level Challenges: The paper highlights that performance can be significantly impacted by differences in character sets and feature representations between language pairs. For instance, languages with vastly different alphabets or complex feature sets pose greater challenges.
  • Impact of Data Imbalance: The results underscore the difficulty of achieving high accuracy on low-resource languages without effective transfer learning strategies, as demonstrated by the poor performance of System 1 trained solely on low-resource data.

Key Results

The paper presents results in terms of accuracy for various language pairs. While specific aggregate numbers are not the primary focus, the tables (Table 2 and Table 3) provide detailed accuracy scores for System 1 (trained on low-resource data only), System 1 (trained on both high and low-resource data, referred to as "tune"), System 2 (transfer learning), and the organizers' baseline.

  • System 2 generally outperforms System 1 (tune) on many language pairs, indicating the benefit of the transfer learning approach.
  • A version of System 1 trained only on low-resource data performed the worst, confirming the necessity of leveraging high-resource data.
  • The authors note that minor tooling mistakes affected the surprise language pairs after the submission deadline, meaning the reported results might not fully reflect their best possible performance.
  • For the "bashkir-khakas" language pair, System 2 achieved a validation loss of approximately 0.2, while System 1 achieved around 0.4, demonstrating a significant improvement in learning efficiency.

Limitations and Future Work

  • Tooling Mistakes: The authors acknowledge that minor tooling errors impacted the results for the surprise language pairs, which were not corrected after the submission deadline.
  • Hyperparameter Optimization: While some hyperparameter tuning was done for System 2, the authors suggest that further optimization (e.g., using cross-validated grid search) could lead to minor improvements.
  • Character Set Differences: The paper identifies differences in character sets and feature sets as a major impediment to performance. They suggest that a phonological mapping to a phonetic alphabet could potentially mitigate these issues for character-set-related problems.
  • Future Work: The authors' ongoing work aims to improve their morphology system for their Natural Language Generation SaaS. They evaluate different solutions for various language and part-of-speech combinations, indicating a pragmatic approach to deploying the best-performing system for each scenario.

Summary updated

BibTeX
@inproceedings{madsack2019ax,
  author = {Andreas Madsack and Robert Weißgraeber},
  title = {{AX Semantics’ Submission to the SIGMORPHON 2019 Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure