Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data

Fahad AlGhamdi, Mona Diab

TL;DR

This paper tackles the challenge of Part-of-Speech (POS) tagging for code-switched (CS) text, where speakers mix languages within a single utterance. They explore how different pre-trained word embedding strategies and neural network architectures impact POS tagging accuracy across four language pairs, finding that related languages benefit from multilingual embeddings, while distant ones benefit from joint POS and language identification tasks.

What Problem Does This Paper Address?

Code-switching (CS) is a common linguistic phenomenon where individuals seamlessly blend two or more languages or dialects within a single conversation. While state-of-the-art Natural Language Processing (NLP) technologies are typically designed for monolingual data, processing CS data presents a significant challenge. Specifically, this paper focuses on the problem of Part-of-Speech (POS) tagging for intra-sentential CS, where languages are mixed within the same sentence. Traditional NLP models struggle with CS data due to issues like out-of-vocabulary words from different languages, hybrid grammatical structures, and ambiguous words that exist in multiple lexicons. This degradation in performance is particularly problematic for downstream NLP tasks that rely on accurate POS tagging, such as parsing, sentiment analysis, and machine translation. The scarcity of annotated CS data further exacerbates this challenge.

Methods

The authors propose and evaluate several approaches to improve POS tagging for CS data, primarily focusing on leveraging pre-trained word embeddings and neural network architectures.

  • Neural Network Architectures: They employ three main neural network models:
  • BiLSTM-CRF Tagger: A standard bidirectional Long Short-Term Memory (BiLSTM) network with a Conditional Random Field (CRF) layer, initialized with pre-trained embeddings.
  • MTL-POS Tagger: A multi-task learning model designed to jointly learn POS tagging for related language pairs (e.g., MSA-EGY and MSA-LEV). It uses two CRF output layers, one for each language pair, sharing a common hidden layer.
  • MTL-POS+LID Tagger: Another multi-task learning model that jointly learns POS tagging and language identification (LID). This model has two output layers, one for POS tags and one for LID tags.
  • Pre-trained Word Embeddings: They explore various strategies for generating pre-trained word embeddings:
  • Monolingual Embeddings (Baseline): Embeddings trained separately on monolingual corpora for each language.
  • Merged Bilingual Embeddings: * Merged Multilingual Embeddings (Pivot): Merging corpora from language pairs sharing a common "pivot" language (e.g., MSA for MSA-EGY and MSA-LEV) to create a single embedding space.
  • Projected Bilingual Embedding: Using the MUSE model to project embeddings from two languages into a shared space, leveraging unsupervised techniques.
  • Code-Switched Embeddings: * Combined Filtered Monolingual Corpora (CFM): Training on a combination of monolingual corpora, primarily capturing inter-sentential CS.
  • Pure Code Switched Corpora (PCS): Training on actual code-switched data, capturing intra-sentential CS.
  • Pseudo Combined Monolingual and CS Corpora (PseudoCS): Combining CFM and PCS corpora to capture both types of CS.

The FastText tool is used for training word embeddings due to its ability to handle subword information, which is beneficial for noisy social media data and morphologically rich languages.

Datasets

The study utilizes several datasets for training and evaluation across four language pairs:

  • MSA-EGY:
    • Evaluation: LDC Egyptian Arabic Treebanks 1-5 (ARZ1-5).
  • Training (Embeddings): Crawled Egyptian tweets, Egyptian online news commentary corpus, Egyptian tweets from CS shared tasks, MSA Gigaword, and LDC Arabic tree bank corpora.
  • MSA-LEV:
    • Evaluation: Curras Corpus of Palestinian Arabic.
  • Training (Embeddings): Tweets from Levantine public figures, Levantine and MSA raw textual data from online news commentary, weblogs, and LDC Arabic tree bank corpora.
  • SPA-ENG:
    • Evaluation: Miami Bangor (MB) corpus.
  • Training (Embeddings): English Universal Dependencies (UD) corpus, Spanish UD corpus, and other English monolingual data.
  • HIN-ENG:
    • Evaluation: Hindi-English Code switching treebank (CS tweets).
  • Training (Embeddings): English UD corpus, Hindi UD corpus, and other Hindi monolingual data.

The Universal POS tag set (Petrov et al., 2011) is used for all language pairs.

Key Insights and Ideas

  • Language Relatedness Matters: The effectiveness of multilingual embeddings and joint POS tagging is highly dependent on the relatedness of the languages. Closely related languages (like MSA-EGY and MSA-LEV) benefit significantly from these approaches, while distant language pairs (like SPA-ENG and HIN-ENG) do not benefit as much, and can even be hindered by the added "noise."
  • Joint POS and LID for Distant Languages: For language pairs that are linguistically distant, jointly learning POS tagging and language identification (MTL-POS+LID Tagger) proves more beneficial than solely relying on multilingual embeddings. This suggests that explicitly identifying language boundaries aids in processing these more challenging combinations.
  • Code-Switched Embeddings Outperform: Embeddings trained on actual code-switched data (PCS and PseudoCS) generally lead to better POS tagging performance compared to purely monolingual or merged bilingual embeddings, especially for intra-sentential CS.
  • Short CS Fragments are Tricky: The presence of short code-switching fragments (insertions of a few words) within sentences poses a significant challenge for even the best classifiers, leading to misclassifications across most models and language pairs.
  • FastText's Subword Advantage: FastText's ability to leverage subword (n-gram) information is crucial for handling noisy social media data and morphologically rich languages, making it a suitable choice for training embeddings in this context.

Key Results

  • MSA-EGY and MSA-LEV: The best performing models (MTL-POS Tagger + Merged Bilingual PseudoCS embeddings) achieved accuracies of 92.90% and 92.92%, respectively. The MSA-EGY result outperforms the state-of-the-art by approximately 2%.
  • SPA-ENG: The BiLSTM-CRF + Merged Bilingual PseudoCS embeddings achieved an accuracy of 96.55%, which is comparable to the state-of-the-art system (96.63%).
  • HIN-ENG: The best model achieved an accuracy of 86.01%, which underperforms the state-of-the-art system (91.90%).
  • LID Task: For the language identification task, all proposed models underperformed existing state-of-the-art systems.

Limitations and Future Work

  • Limitations:
    • The study found that short CS fragments are particularly difficult to classify, indicating a limitation in current models' ability to handle fine-grained code-switching.
  • While the models show promise, the HIN-ENG results underperform the state-of-the-art, suggesting room for improvement in handling more distant language pairs.
  • The LID task performance was not competitive with existing systems.
  • Future Work:
    • Investigate the theoretical aspects of word embedding learning for CS data.
  • Apply the proposed word embedding strategies to other downstream NLP tasks, such as segmentation and parsing.
  • Further explore how to effectively handle short code-switching fragments.

Summary updated

Abstract

Linguistic Code Switching (CS) is a phenomenon that occurs when multilingual speakers alternate between two or more languages/dialects within a single conversation. Processing CS data is especially challenging in intra-sentential data given state-of-the-art monolingual NLP technologies since such technologies are geared toward the processing of one language at a time. In this paper, we address the problem of Part-of-Speech tagging (POS) in the context of linguistic code switching (CS). We explore leveraging multiple neural network architectures to measure the impact of different pre-trained embeddings methods on POS tagging CS data. We investigate the landscape in four CS language pairs, Spanish-English, Hindi-English, Modern Standard Arabic- Egyptian Arabic dialect (MSA-EGY), and Modern Standard Arabic- Levantine Arabic dialect (MSA-LEV). Our results show that multilingual embedding (e.g., MSA-EGY and MSA-LEV) helps closely related languages (EGY/LEV) but adds noise to the languages that are distant (SPA/HIN). Finally, we show that our proposed models outperform state-of-the-art CS taggers for MSA-EGY language pair.
BibTeX
@inproceedings{alghamdi2019leveraging,
  author = {Fahad AlGhamdi and Mona Diab},
  title = {{Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure