CBNU System for SIGMORPHON 2019 Shared Task 2: a Pipeline Model

Uygun Shadikhodjaev, Jae Sung LeeACL 2019

TL;DR

This paper presents a novel pipeline model for morphological analysis and lemmatization, combining a Transformer-based sequence-to-sequence model for lemmatization with a biaffine attention-based BiLSTM for morphological analysis. The system achieved competitive results in the SIGMORPHON 2019 Shared Task 2, demonstrating the effectiveness of this two-stage approach, particularly the benefit of incorporating lemma information into the morphological analysis.

What Problem Does This Paper Address?

The paper tackles the challenge of morphological analysis and lemmatization, crucial tasks in Natural Language Processing (NLP) for understanding word forms and their base meanings. Lemmatization involves mapping a word to its dictionary form (lemma), which can be an "m-to-n" mapping (different lengths). Morphological analysis, on the other hand, assigns a set of tags (MSD - morphosyntactic description) to a word based on its context within a sentence, essentially a classification task. The SIGMORPHON 2019 Shared Task 2 specifically aimed to address these tasks across a wide range of languages, including many low-resource ones, posing a significant challenge due to data scarcity and linguistic diversity. The authors aim to build a robust system that can handle these complexities effectively.

Methods

The CBNU system employs a two-stage pipeline:

  1. Lemmatizer: This component uses a Transformer-based encoder-decoder architecture, similar to those used in neural machine translation. It treats lemmatization as a sequence-to-sequence task, mapping an input word (as a sequence of characters) to its lemma (also as a sequence of characters). This approach is justified by the success of Transformers in translation tasks. The model is trained on character sequences, without pre-trained word embeddings, and uses a transformer_tiny configuration for efficiency.
  2. MSD Tagger: This component takes the original word and its predicted lemma as input and performs morphological analysis. It's framed as a classification task. The authors adapt a modified dependency parser architecture (based on Dozat et al., 2017) that uses biaffine attention. The input to this model is an elementwise addition of four embeddings: pre-trained, trainable, character-level, and lemma embeddings. The model then passes these through BiLSTM layers, followed by a Multi-Layer Perceptron (MLP) and biaffine attention layers to predict the MSD tag.

A key innovation is the pipelining order: they first perform lemmatization and then use the predicted lemma as an additional input for the MSD tagger. This contrasts with some baseline approaches that might first predict MSD tags and then use them for lemmatization.

Datasets

The SIGMORPHON 2019 Shared Task 2 dataset is used, which comprises initial datasets from over 60 languages, with an additional nine surprise languages added later. The paper highlights the significant variation in dataset sizes, ranging from very low-resource languages like Akkadian (80 sentences) to high-resource languages like Czech PDT (70,330 sentences). This diversity, especially the presence of low-resource languages, is a key challenge addressed by the system.

Key Insights and Ideas

  • Pipeline Order Matters: The decision to perform lemmatization before morphological analysis and to feed the predicted lemma into the MSD tagger proved beneficial. Experiments showed that this order improved MSD tagging accuracy, suggesting that lemma information is crucial for accurate morphological analysis, especially for morphologically rich languages.
  • Transformer for Lemmatization: Leveraging the Transformer architecture, successful in machine translation, for character-level sequence-to-sequence lemmatization is a sound approach, especially given its ability to handle variable-length inputs and outputs.
  • Multi-faceted Embeddings for MSD Tagging: Combining pre-trained, trainable, character-level, and lemma embeddings provides a rich representation for the MSD tagger, allowing it to capture various aspects of a word's form and meaning.
  • Character-Level Embeddings: The use of character-level embeddings, processed by a unidirectional LSTM, helps in handling out-of-vocabulary words and capturing sub-word information, which is vital for morphologically complex languages.

Key Results

The CBNU system achieved competitive results in the SIGMORPHON 2019 Shared Task 2:

  • Lemmatization: Ranked fifth among twelve systems.
  • Morphological Accuracy: Ranked sixth among twelve systems.
  • The paper notes that incorporating lemma information into MSD tagging demonstrated "considerable improvements over the baseline."
  • Ablation studies (though not fully detailed in the provided text) indicated that removing lemma information for MSD tagging led to a decrease in accuracy, particularly for morphologically complex languages like Korean.

Limitations and Future Work

  • Time Constraints: The study was conducted under time constraints, limiting the extent of experiments. For instance, experiments for MSD tagging without lemma information were performed only on English-PUD and Korean-Kaist.
  • Future Work Suggestion: The authors suggest exploring how incorporating the output of MSD tagging back into the lemmatization process might affect lemma accuracy. This hints at a potential for more integrated or iterative models.
  • Low-Resource Language Challenges: While the system performed well, the inherent challenges of low-resource languages (lack of pre-trained word vectors, small datasets) are acknowledged. The paper mentions using related languages' word vectors as a substitute, but this is a general strategy rather than a specific solution presented in detail.

Summary updated

BibTeX
@inproceedings{shadikhodjaev2019cbnu,
  author = {Uygun Shadikhodjaev and Jae Sung Lee},
  title = {{CBNU System for SIGMORPHON 2019 Shared Task 2: a Pipeline Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure