CUNI–Malta system at SIGMORPHON 2019 Shared Task on Morphological Analysis and Lemmatization in context: Operation-based word formation

Ronald Cardenas, Claudia Borg, Daniel ZemanACL 2019

TL;DR

This paper introduces a novel operation-based neural transducer for morphological analysis and lemmatization. By treating word formation as a sequence of operations rather than character-by-character processing, their model aims to capture more expressive linguistic patterns, particularly for morphologically rich languages. While not outperforming the baseline on all metrics, their approach demonstrates an ability to associate interpretable operations with fine-grained linguistic features.

What Problem Does This Paper Address?

Morphological analysis and lemmatization are crucial tasks in natural language processing, especially for morphologically rich languages where words can have many forms derived from a single lemma. Traditionally, these tasks were handled by finite-state transducers. More recently, neural sequence-to-sequence models have been applied, but many still operate by processing input character by character. This character-level processing can struggle with capturing long-range dependencies within words and across sentences, which are vital for accurate analysis. The SIGMORPHON 2019 Shared Task specifically focuses on these challenges, including cross-lingual transfer and in-context analysis, highlighting the need for more robust and interpretable models.

Methods

The core innovation of this paper lies in their operation-based word formation process. Instead of processing input character by character, their neural transducer generates a sequence of operations that transform a word form into its lemma. These operations are designed to be more expressive and interpretable, encoding not just the action (insert, delete, substitute, transpose) but also the position within the word and the segment of characters involved.

To manage the complexity of these operations, they employ a strategy inspired by Byte-Pair Encoding (BPE). This involves merging frequent adjacent operations to reduce the space of valid operations, making them more learnable. The resulting operations are then sorted to ensure a consistent processing order (prefix, then inner-word, then suffix).

The system consists of two pipelined models:

  1. Lemmatization Model: This model treats lemmatization as a language modeling task over the generated action sequences. It uses an RNN with an LSTM cell to encode the action sequence, incorporating the original word form as an initial input.
  2. Morphological Tagging Model: This model takes the encoded action representations from the lemmatizer and uses a bidirectional LSTM to predict the morpho-syntactic description (MSD) label for each token.

Datasets

The models were trained and evaluated on data from the SIGMORPHON 2019 Shared Task. This task utilizes data from Universal Dependencies v.2.3 treebanks, covering 66 languages. Feature bundles were translated into the UniMorph tagset using a specified mapping strategy. The paper mentions using the development set of Spanish (es_ancora) for hyperparameter tuning.

Key Insights and Ideas

  • Operation-based transduction is more expressive: By defining operations that act on segments of characters and include positional information, the model can potentially capture more nuanced linguistic transformations than character-level models.
  • BPE-inspired merging simplifies operations: The strategy of merging frequent adjacent operations effectively reduces the action space, making the model more tractable and learnable, while still retaining interpretability.
  • Interpretable operations correlate with linguistic features: The analysis shows that the model can learn associations between its interpretable operation labels and fine-grained morpho-syntactic features, offering insights into how the model "thinks" about word formation. For example, the association of a plural marker with a deletion operation.
  • Character-level processing can be a bottleneck: The authors hypothesize that their model's underperformance on some treebanks stems from the complexity and sparsity of the operation set, particularly when dealing with languages that have very little morphology or unusual script representations.

Key Results

  • The proposed models underperform the shared task baseline on most treebanks in terms of lemmatization accuracy and MSD-F1 scores.
  • However, the model shows improvements over the baseline on specific languages like Tagalog, Chinese, Cantonese, and Amharic.
  • In morphological tagging, an error increase was observed, but significant error decreases were noted for Russian and Finnish.
  • The analysis of action-feature co-occurrences revealed meaningful associations, such as the link between a plural marker and a deletion operation, and the negation prefix "ne" with verb negation.
  • For languages with little morphology (e.g., Chinese, Vietnamese), the lemmatizer exhibits a bias towards copying the input word form, achieving very high accuracy but potentially not learning true morphological transformations.

Limitations and Future Work

  • Fixed Gold Action Sequences: The reliance on pre-defined "gold" action sequences means the model cannot benefit from alternative valid sequences that might exist, and it is penalized for predicting them.
  • Monotonic Correspondence Assumption: The model assumes a near-monotonic alignment between input and output characters and a same-script condition. This assumption is violated in cases like Arabic-PUD (romanized lemmas) and Akkadian-PISANDUB (mixed writing systems), leading to poor performance.
  • Bias towards Copying: For morphologically poor languages, the model tends to copy the input, which, while accurate, might not reflect genuine morphological analysis.
  • Input Representation: The authors hypothesize that the input representation (action sequences) might be too fine-grained and sparse, even after BPE merging, contributing to underperformance.

Future work could explore more flexible ways to handle non-monotonic alignments and mixed-script data, as well as investigate alternative methods for generating or learning action sequences that are less prone to copying bias.

Summary updated

BibTeX
@inproceedings{cardenas2019cunimalta,
  author = {Ronald Cardenas and Claudia Borg and Daniel Zeman},
  title = {{CUNI–Malta system at SIGMORPHON 2019 Shared Task on Morphological Analysis and Lemmatization in context: Operation-based word formation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure