Joint Learning Model for Low-Resource Agglutinative Language Morphological Tagging

Gulinigeer Abudouwaili, Kahaerjiang Abiderexiti, Nian Yi, Aishan WumaierACL 2023

TL;DR

This paper introduces a novel joint learning model for morphological tagging in low-resource agglutinative languages. By jointly training part-of-speech (PoS) and morphological feature (MF) tagging and incorporating a sophisticated label relationship network, the model significantly improves tagging accuracy, addressing a critical gap in NLP for under-resourced languages.

What Problem Does This Paper Address?

Morphological tagging, the process of identifying a word's grammatical properties (like part-of-speech and features like tense or case), is crucial for understanding language. However, for agglutinative languages (where words are formed by adding many suffixes to a root, like in Turkish or Finnish), this task is particularly challenging due to data scarcity. Existing methods often rely on expert knowledge, struggle with contextual information, and suffer from error propagation. While high-resource languages benefit from deep learning, low-resource languages are often limited to rule-based or transfer learning approaches that have their own drawbacks. This paper aims to bridge this gap by developing a robust deep learning model specifically for low-resource agglutinative languages.

Methods

The proposed model tackles morphological tagging by dividing it into two subtasks: Part-of-Speech (PoS) tagging and Morphological Feature (MF) tagging.

  • Joint Learning & Fusion Mechanism: Instead of treating these as separate tasks, the model jointly trains them. This is crucial because PoS can disambiguate morphological features, and vice-versa. A "fusion mechanism" is employed in the middle layers to allow these two tasks to influence each other, transforming direct error propagation into a more indirect, constrained interaction.
  • Multi-dimensional Input Representation: The model enriches word representations by combining three types of embeddings: word embeddings (from pre-trained language models like CINO for Uyghur and XLM-RoBERTa for others), morphological embeddings (learned via BiLSTM), and local context embeddings.
  • Morphological Feature Relation Network (MFRN): This is a key innovation. It explicitly models relationships between MF labels.
  • Label Embeddings: MF labels are embedded into a vector space.
  • Adjacency Matrices: Two types of adjacency matrices are constructed:
  • Relevant Label Adjacency (RLA): Captures universal relationships between MF labels across all languages.
  • Irrelevant Label Adjacency (ILA): Captures language-specific relationships, constructed for each of the four target languages.
  • Graph Convolutional Networks (GCNs): GCNs are used to learn complex, higher-order relationships between MF labels, guided by these adjacency matrices. This allows the model to capture subtle dependencies that statistical methods might miss.
  • Inference Layer: The PoS tagger uses a Conditional Random Field (CRF) for sequence labeling, while the MF tagger utilizes the MFRN output.

Datasets

The model was evaluated on four low-resource agglutinative languages from the Universal Dependencies (UD) treebank:

  • Uyghur (Ug)
  • Kazakh (Kz)
  • Tatar (Tt)
  • Yakut (Yk)

The data was converted to the UniMorph format and split into training, validation, and testing sets in an 8:1:1 proportion.

Key Insights and Ideas

  • Joint training is more than the sum of its parts: The paper demonstrates that jointly training PoS and MF tagging, with a fusion mechanism, significantly reduces error propagation and improves overall accuracy compared to independent models.
  • Label relationships matter, especially for low-resource languages: Explicitly modeling the relationships between morphological features using GCNs and adjacency matrices is crucial for overcoming data scarcity. This allows the model to leverage linguistic structure even with limited examples.
  • Language-specific nuances are important: While universal relationships are captured, the inclusion of language-specific irrelevant label adjacency matrices allows the model to adapt to the unique morphological structures of each language.
  • Context and morphology are intertwined: The experiments highlight that both word formation (morphology) and local context are vital for accurate tagging, and their combination leads to better performance.

Key Results

The proposed model achieved state-of-the-art results on the UKTY datasets.

  • Overall Performance: The model achieved an average accuracy of 85.29% and an average F1 score of 92.61% across the four languages.
  • Comparison to Baselines: The model outperformed several strong baselines, including Neural Tagger, Multi-Team, Morpheus, UDify, UDPipe, and COMBO. For example, on the Uyghur dataset, the model achieved a 95.52% F1 score, significantly higher than many others.
  • Impact of Fusion Mechanism: Removing the fusion mechanism led to a decrease in F1 score by 5.44% and accuracy by 5.71% on average, underscoring its importance.
  • Impact of Label Relationships: Ablation studies showed that adding different types of label relationships (initial, irrelevant, pre-trained, reconstructed) generally improved performance, with the combination of all relationships yielding the best results.
  • Error Analysis: The model demonstrated robustness in predicting both short and long morphological labels, showing less sensitivity to label length and overlap compared to other models.

Limitations and Future Work

  • Noise Sensitivity: The model's reliance on label relationships means that noisy data or incorrect label relationships can negatively impact performance.
  • Training Time: Learning pre-trained label relationships requires more training time compared to models that don't explicitly learn these relationships.

The authors suggest future work could focus on optimizing relationship representation, improving threshold selection abilities, and further enhancing model performance.

Summary updated

BibTeX
@inproceedings{abudouwaili2023joint,
  author = {Gulinigeer Abudouwaili and Kahaerjiang Abiderexiti and Nian Yi and Aishan Wumaier},
  title = {{Joint Learning Model for Low-Resource Agglutinative Language Morphological Tagging}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure