Morphology is not just a naive Bayes – UniMelb Submission to SIGMORPHON 2022 ST on Morphological Inflection

Andreas Sherbakov, Ekaterina VylomovaNAACL 2022

TL;DR

This paper presents Flexica, a non-neural system for morphological inflection that extracts transformation patterns from lemma-inflected form pairs. The system achieved strong performance, outperforming the non-neural baseline, but highlights the limitations of simple pattern-matching approaches when dealing with complex linguistic phenomena and sparse data.

What Problem Does This Paper Address?

Morphological inflection, the process of changing a word's form to indicate grammatical properties like tense or number, is a fundamental aspect of language. While neural models have shown great success in high-resource languages, their performance drops significantly in low-resource settings where paradigms are less represented. The SIGMORPHON 2022 Shared Task specifically aimed to test systems' ability to generalize to unseen lemmas and feature combinations, addressing the challenge of robust morphological inflection in diverse and often data-scarce scenarios. This paper tackles the problem of creating a system that can effectively learn and apply these inflectional rules, particularly when faced with limited training data and the need to generalize beyond observed examples.

Methods

The Flexica system is a non-neural approach that operates by extracting transformation patterns from aligned lemma-inflected form pairs in the training data. For each inflection, it identifies common substrings between the lemma and its inflected form and represents these as patterns, often using wildcards to generalize. The system generates multiple pattern variants for each training sample, considering different levels of abstraction and potential wildcard placements.

Key aspects of their method include:

  • Pattern Extraction: Identifying longest common substrings and replacing them with wildcards to create transformation patterns (e.g., lemma_pattern, tag -> form_pattern).
  • Pattern Refinement: Partially reverting wildcard groups to their concrete characters to capture more specific rules and exceptions.
  • Trie Database: Storing all unique patterns in a trie data structure for efficient lookup.
  • Scoring Mechanism: A hierarchical scoring system to select the best transformation pattern for a given lemma-tag pair. This score considers factors like pattern abstractness (number of characters in wildcards), tag set mismatch, a "representative premium" for abstract but correct patterns, and pattern frequency.

Datasets

The system was evaluated on the SIGMORPHON 2022 Shared Task 1 Part 1 dataset, which comprises data for 33 typologically diverse languages. The task data is annotated using the UniMorph schema and is split into training, development, and test sets designed to assess different types of generalization. The dataset includes both "small" (low-resource) and "large" (high-resource) settings, with varying numbers of training instances per language.

Key Insights and Ideas

  • Morphology is More Than Naive Bayes: The paper's title itself suggests that simple probabilistic approaches (like naive Bayes) are insufficient for capturing the nuances of morphological inflection, especially when generalization is required.
  • Pattern Abstraction is Key, But Not Sufficient: The system's success in outperforming the non-neural baseline demonstrates the power of extracting and generalizing transformation patterns. However, the authors found that scores based solely on abstractness and frequency were not enough to handle all cases, particularly those involving complex interactions or exceptions.
  • The Challenge of Sparse Data and Generalization: The shared task's design, with its focus on unseen lemmas and feature combinations, highlighted the inherent difficulty of morphological inflection in low-resource settings. The system's performance, while good, still revealed limitations in handling these scenarios.
  • Rule-and-Exception Hierarchies: The authors hypothesize that languages often have a hierarchy of rules and exceptions, and their scoring mechanism attempts to account for this. However, they acknowledge that their current definition of "concreteness" is limited.

Key Results

The Flexica system outperformed the non-neural baseline across many languages in both small and large settings. The paper presents accuracy tables (Table 1 and Table 2) showing the system's performance. For instance, in the large setting, Flexica's submitted accuracy ("Sb.") often surpassed the non-neural baseline ("BL") and sometimes approached the best neural systems ("max"). The authors also explored modifications to their scoring mechanism, such as increasing the penalty for impure patterns or adding a bonus for exact tag matches, but these generally had minor impacts on overall accuracy.

Limitations and Future Work

The authors acknowledge several limitations:

  • Insufficient Scoring Criteria: The current scoring mechanism, while hierarchical, does not fully account for all possible combinations of string segments and morphosyntactic features, leading to errors in a proportion of cases.
  • Pattern Proliferation: The system generates a large number of patterns, leading to "noisy" patterns that do not add significant value and can hinder the consideration of longer, more informative subsequences.
  • Lack of Affix Concatenation Mechanism: This limitation specifically affected performance in agglutinative languages like Turkish and Hungarian, where the system struggled with combining multiple affixes.
  • Tag Combination Sparsity: The extreme sparsity of grammatical tag combinations in the dataset made it difficult for models to learn robust tag interchangeability.

Future work suggested includes:

  • Developing more sophisticated pattern matching mechanisms, potentially incorporating "soft attention" features.
  • Exploring better classification models.
  • Investigating ways to reduce the excessive number of generated patterns.
  • Developing mechanisms to handle affix concatenation more effectively.
  • Exploring more advanced techniques for dealing with sparse tag combinations, potentially moving beyond simplistic "edit distance" approaches.

Summary updated

BibTeX
@inproceedings{sherbakov2022morphology,
  author = {Andreas Sherbakov and Ekaterina Vylomova},
  title = {{Morphology is not just a naive Bayes – UniMelb Submission to SIGMORPHON 2022 ST on Morphological Inflection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure