AIDA: Identifying Code Switching in Informal Arabic Text

Heba Elfardy, Mohamed Al-Badrashiny, Mona T. DiabCodeSwitch @ EMNLP 2014 · OpenReview

TL;DR

This paper introduces AIDA, an updated system for identifying code-switching between Modern Standard Arabic (MSA) and various Arabic dialects in informal text. By leveraging a sophisticated morphological analyzer (MADAMIRA) and language models, AIDA achieves high accuracy in classifying words, crucial for downstream NLP tasks in the Arabic language.

What Problem Does This Paper Address?

Arabic exhibits diglossia, meaning it has a standard form (MSA) used in formal contexts and numerous dialectal variants (DA) used in everyday communication. Informal Arabic text, especially online, frequently mixes MSA and DA within the same sentence or even word. This code-switching poses a significant challenge for Natural Language Processing (NLP) tasks like machine translation, sentiment analysis, and information retrieval, as standard NLP tools often struggle with this linguistic variation. Existing systems have attempted to tackle this, but there's a need for robust methods that can accurately identify and classify these dialectal shifts at a word level.

Methods

The core of AIDA is a pipeline that processes informal Arabic text to identify code-switching. The system utilizes two main components:

  • Language Models (LM): These models are trained on large corpora to predict the likelihood of word sequences.
  • Morphological Analysis and Disambiguation: The paper highlights the use of MADAMIRA, a tool that performs tokenization (breaking down words into morphemes), lemmatization, and Part-of-Speech (POS) tagging. A key advantage of MADAMIRA is its contextual disambiguation, which helps resolve ambiguities in word analysis.

The system employs a Combiner module that uses a set of rules to assign the final tag to each word based on the outputs from the LM and MADAMIRA, as well as a named entity list. The rules prioritize certain tags (e.g., "other" for punctuation/numbers, "NE" for named entities) and then fall back to dialect identification based on LM predictions or morphological analysis.

Two preprocessing schemes are explored:

  1. Basic: Standard text cleaning, replacing punctuation, URLs, and numbers with keywords.
  2. Tokenized: In addition to basic preprocessing, this scheme uses MADAMIRA to tokenize clitics and affixes, allowing for finer-grained analysis of word morphology.

Datasets

  • Shared-task's training data (STT): A corpus of 119,326 words collected from Twitter, manually annotated at the token level. This was split into a training set (STT-Tr) and a development set (STT-Dev).
  • Web-log training data (WLT): A large corpus of 8 million words from web logs, weakly labeled by forum dialect (MSA or EDA).
  • Test Sets: Three blind test sets were used for evaluation:
  • Test1: 54,732 words from 2,363 tweets.
  • Test2: 32,641 words from 1,777 tweets.
  • Surprise: 12,017 words from Arabic commentaries, representing a different genre.

The authors also mention using the ANERGazet for named entity recognition, which consists of gazetteers for locations, people, and organizations.

Key Insights and Ideas

  • Tokenization is Crucial: The "Tokenized" preprocessing scheme, which breaks down words into their constituent morphemes, consistently outperforms the "Surface Form" setup. This highlights the importance of analyzing word structure beyond just the surface form for accurate dialect identification.
  • MADAMIRA's Value: The integration of MADAMIRA significantly enhances the system by providing rich morphological information and contextual disambiguation, which is vital for handling the complexities of Arabic dialects.
  • Named Entity Recognition Robustness: AIDA demonstrates strong performance in identifying named entities, outperforming other systems in this specific task. This suggests the NER component is particularly robust.
  • Confusability between MSA and DA: The error analysis reveals that the highest confusion occurs between MSA ("lang1") and dialectal Arabic ("lang2") classes, underscoring the inherent difficulty in distinguishing between them, especially when context is ambiguous.

Key Results

The paper reports impressive results on the shared task datasets:

  • Token-level FB=1 scores: AIDA achieved an average token-level FB=1 score of 93.6% on the first test set, 77.7% on the second, and 80.1% on the surprise genre set.
  • Tweet-level FB=1 scores: The system achieved tweet-level FB=1 scores of 4.4%, 36%, and 27.7% on the same test sets, respectively.
  • Overall Performance: AIDA's best configuration (Tokenized-2) achieved a weighted-average token-level FB=1 score of 77.6% on the development set. On the test sets, AIDA outperformed other participating systems in token-level evaluation and was second only to CMU in tweet-level classification.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the size of the STT corpus is small compared to WLT, which can introduce noise. They addressed this by adjusting STT labels and duplicating STT-Tr.
  • The confusion between "lang1" and "lang2" is a persistent challenge, especially for words that can function in both MSA and DA depending on context.
  • The system's performance on the "ambig" tag is relatively low, indicating challenges in identifying truly ambiguous words.
  • Future Work:
    • The authors plan to extend AIDA to handle other Arabic dialects beyond Egyptian, such as Levantine, Iraqi, and Moroccan Arabic.
  • They also aim to adapt the system to other genres of informal text.

Summary updated

BibTeX
@inproceedings{elfardy2014aida,
  author = {Heba Elfardy and Mohamed Al-Badrashiny and Mona T. Diab},
  title = {{AIDA: Identifying Code Switching in Informal Arabic Text}},
  booktitle = {CodeSwitch @ EMNLP},
  year = {2014},
  url = {https://openreview.net/forum?id=7P3k5PCTna},
}
Figure