SIGMORPHON 2022 Task 0 Submission Description: Modelling Morphological Inflection with Data-Driven and Rule-Based Approaches

Tatiana Merzhevich, Nkonye Gbadegoye, Leander Girrbach, Jingwen Li, Ryan Soh-Eun ShimNAACL 2022

TL;DR

This paper explores two approaches to morphological inflection: a modified neural encoder-decoder and hand-crafted finite-state transducers (FSTs). While the neural approach underperformed, the FSTs achieved state-of-the-art results on low-resource languages, highlighting the continued value of linguistic expertise in computational morphology, especially for endangered languages.

What Problem Does This Paper Address?

The paper tackles the challenge of morphological inflection generation, specifically in the context of the SIGMORPHON 2022 Shared Task. This task involves predicting inflected word forms given a lemma and a set of morphological tags. A key challenge in this area is handling typologically diverse languages and dealing with low-resource scenarios where large amounts of training data are unavailable. Traditional data-driven methods, while powerful, can struggle with interpretability and may not always be the most effective for languages with limited digital resources or complex morphological systems. The paper aims to compare a data-driven approach with a rule-based approach (FSTs) to understand their respective strengths and weaknesses in this domain.

Methods

The authors present two main methodologies:

  • Data-Driven Approach (Modified Encoder-Decoder): This approach is a modification of a standard encoder-decoder model. Instead of predicting characters directly, the model predicts transitions between states in a directed graph. This graph represents allowed character sequences, aiming to improve interpretability and control over generated forms by disallowing illegal transitions. The graph is constructed by aligning paradigms in the training data.
  • Rule-Based Approach (Finite-State Transducers - FSTs): For specific languages (Chukchi, Upper Sorbian, Kholosi), the authors built morphological analyzers using Foma, a finite-state compiler. These FSTs are based on lexicons and hand-crafted rules derived from regular expressions. The development of these FSTs leveraged linguistic knowledge and external resources like the UniMorph dataset.

Datasets

The primary dataset used is the UniMorph 2.0 dataset, which provides lists of lemmas, word forms, and universal feature schemas for various languages. This dataset was used for both training and evaluation across different languages and resource settings (low-resource and high-resource scenarios). The authors also mention using publicly available data from UniMorph for constructing the FSTs, which they acknowledge might have led to a violation of the train/test set separation.

Key Insights and Ideas

  • FSTs Excel in Low-Resource Settings: The most striking finding is the superior performance of hand-crafted FSTs on low-resource languages compared to the data-driven approach. This suggests that linguistic expertise can compensate for data scarcity.
  • Interpretability vs. Performance Trade-off: The modified encoder-decoder aimed for better interpretability by using a graph structure, but this came at the cost of performance, underperforming the baseline. This highlights a common tension in machine learning: interpretability often requires sacrificing some predictive power.
  • Linguistic Expertise Remains Crucial: The success of the FSTs strongly suggests that for complex morphological systems and under-documented languages, human linguistic knowledge is still indispensable and can outperform purely data-driven methods.
  • Potential for Hybrid Approaches: While not explicitly implemented as a hybrid system in this submission, the paper implicitly points towards the benefits of combining data-driven learning with rule-based constraints or knowledge injection.

Key Results

  • The data-driven approach (modified encoder-decoder) underperformed the baseline for almost all languages. Its overall accuracy results are presented in Table 3 in the appendix.
  • The FST-based approach significantly outperformed other systems on the languages for which they were developed (Chukchi, Upper Sorbian, Kholosi), achieving high accuracy scores (e.g., 96.667 for Kholosi, 83.750 for Upper Sorbian, 19.565 for Chukchi, as per Table 2).
  • The authors note that their FSTs might have been influenced by publicly available UniMorph data, potentially leading to an unfair advantage.

Limitations and Future Work

  • Data-Driven Approach Performance: The primary limitation of the data-driven approach is its underperformance compared to the baseline, suggesting the modification did not yield the desired improvements.
  • FST Data Contamination: The authors acknowledge that the creation of FSTs might have inadvertently used publicly available data from UniMorph that was also part of the test set, potentially inflating their performance.
  • Future Work:
    • Expanding the FSTs with more sophisticated graph structures and edge constraints.
  • Developing more expressive auxiliary graphs for the data-driven approach.
  • Exploring extensions to FSTs, such as incorporating guessers for assigning possible stems and part-of-speech tags.
  • Further research into more interpretable encoder-decoder models.

Summary updated

BibTeX
@inproceedings{merzhevich2022sigmorphon,
  author = {Tatiana Merzhevich and Nkonye Gbadegoye and Leander Girrbach and Jingwen Li and Ryan Soh-Eun Shim},
  title = {{SIGMORPHON 2022 Task 0 Submission Description: Modelling Morphological Inflection with Data-Driven and Rule-Based Approaches}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure