A Language-aware Approach to Code-switched Morphological Tagging

Şaziye Betül Özateş, Özlem ÇetinoğluNAACL 2021

TL;DR

Code-switched text, where speakers mix languages, poses a significant challenge for morphological tagging due to differing language structures. This paper introduces LIDVec, a novel approach that integrates language identification (LID) information directly into transformer models, significantly improving morphological tagging accuracy on code-switched data.

What Problem Does This Paper Address?

Morphological tagging, the task of identifying a word's grammatical features (like tense, number, case), is crucial for many Natural Language Processing (NLP) tasks. However, this becomes considerably more complex when dealing with code-switched (CS) data, where speakers fluidly switch between two or more languages within a single utterance. The challenge is amplified when the languages involved have distinct morphological systems. For instance, German has grammatical gender, while Turkish does not, and their case systems differ. Existing NLP models often struggle to handle these cross-linguistic variations, leading to inaccurate morphological analyses. While POS tagging for CS data has received some attention, CS morphological tagging remains largely unexplored. This paper aims to bridge this gap by developing a language-aware approach to improve CS morphological tagging.

Methods

The authors propose LIDVec (Language ID Vectors), a method that integrates language identification (LID) information directly into a transformer-based model. Instead of treating LID as a separate pipeline step or a simple feature, LIDVec creates learnable LID embeddings. These embeddings are then concatenated with the token embeddings generated by a pre-trained transformer model (XLM-R in this case). This combined representation is fed into the model for morphological tagging.

The core idea is that by explicitly providing the model with information about which language a token belongs to, it can better differentiate and learn the language-specific morphological rules. This approach is integrated into the STEPS framework, a multilingual NLP tool for tagging and parsing.

They also compare LIDVec against several baselines:

  • Standalone: A standard STEPS model without any explicit LID integration.
  • DSplit: A pipeline approach where sentences are split into monolingual segments, and separate models are trained for each segment.
  • MTL (Multi-Task Learning): A model trained to perform both morphological tagging and LID tagging simultaneously.

Datasets

The experiments were conducted on the Turkish-German SAGT Treebank (UD version 2.7.1). This treebank is derived from conversational recordings of bilingual Turkish-German speakers and contains normalized transcriptions. It includes five LID labels: TR (Turkish), DE (German), LANG3 (other languages), OTHER (punctuation), and MIXED (for intra-word code-switching).

Key Insights and Ideas

  • Language-Specific Morphology is Key: The paper strongly emphasizes that morphological tagging is inherently language-specific, unlike POS tagging which can often be generalized across languages. This fundamental difference is the root cause of the challenge in CS morphological tagging.
  • LID Integration is Crucial: Explicitly informing the model about the language of each token significantly boosts performance in CS morphological tagging.
  • LIDVec's Robustness: The proposed LIDVec approach, which embeds LID information, proves to be more robust to errors in predicted LIDs compared to the DSplit method. This is because LIDVec's errors are implicitly compensated during end-to-end training, whereas DSplit's errors are propagated through separate models.
  • MTL is Less Effective for this Task: While multi-task learning is a common technique, it was found to be the least effective among the LID integration methods for this specific problem.
  • Impact on Specific Categories: LID integration shows a notable improvement in predicting morphological features for Turkish verbs and nouns, and German pronouns and nouns. It also significantly helps in correctly identifying the Case feature, especially for MIXED tokens.

Key Results

  • The LIDVec model achieved the highest overall accuracy on both development and test sets, outperforming the Standalone model by over 2-3 points.
  • All LID integration methods (DSplit, MTL, LIDVec) outperformed the Standalone model.
  • LIDVec showed a significant improvement in predicting morphological analyses for MIXED tokens, achieving an accuracy of 73.22% on the test set compared to 61.75% for the Standalone model.
  • The paper also reports that LID integration had no significant impact on POS tagging accuracy, as POS tags in the SAGT treebank are universal and language-independent.
  • LIDVec demonstrated better robustness to predicted LIDs compared to DSplit, with only a minor drop in accuracy when using predicted LIDs.

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that the performance of data-driven models is heavily influenced by the amount of training data. While the SAGT Treebank is the only available CS dataset with complete morphological annotation, more data would likely further improve results.
  • LANG3 and OTHER Tokens: The performance on LANG3 (other languages) and OTHER (punctuation) tokens showed less consistent improvement, and further investigation with more data is suggested.
  • Random Initialization Preference: While average initialization of LID embeddings was explored, random initialization was found to be preferable, especially when time constraints are a factor, as it yielded comparable results with sufficient training.
  • Future Work: The authors suggest exploring other LID integration methods and applying the LIDVec approach to other code-switched datasets and tasks that can benefit from language identification. They also highlight the potential for using LIDVec with faster, less computationally expensive LID prediction models.

Summary updated

BibTeX
@inproceedings{ozates2021language,
  author = {Şaziye Betül Özateş and Özlem Çetinoğlu},
  title = {{A Language-aware Approach to Code-switched Morphological Tagging}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure