ZCU-NLP at MADAR 2019: Recognizing Arabic Dialects

Pavel Přibáň, Stephen TaylorACL 2019

TL;DR

This paper presents the ZCU-NLP team's approach to the MADAR 2019 shared task on Arabic dialect identification. They developed two systems for identifying city-level dialects and user country of origin from tweets, achieving competitive results by leveraging language modeling features and simpler machine learning classifiers.

What Problem Does This Paper Address?

Arabic, while having a unified written form, exhibits significant dialectal variation in spoken and informal written contexts. Accurately identifying these dialects is crucial for various Natural Language Processing (NLP) applications, such as machine translation, sentiment analysis, and information retrieval, especially in the context of social media where informal language and dialectal features are prevalent. The MADAR 2019 shared task specifically aimed to address this by focusing on fine-grained dialect identification at both the city and country levels, using real-world data from Twitter.

Methods

The ZCU-NLP team employed two main systems for Subtask-1 (city dialect identification):

  • Tortuous Classifier: This system combined multiple "streams" of features and classifiers. The core idea was to use language model scores (derived from kenlm) as primary features, alongside traditional n-gram features (word and character). These features were fed into a hard voting classifier, which aggregated predictions from a soft voting classifier, a Support Vector Machine (SVM), and a multinomial naive Bayes classifier. They experimented with both word and character n-grams, and importantly, prepared character n-grams by treating each character as a separate "word" to leverage kenlm's capabilities.
  • Neural Network Classifier: This system used a Multilayer Perceptron (MLP) with a single hidden layer. The input features were the probabilities assigned by 26 pre-trained dialect language models. They also explored a more complex architecture involving Bidirectional LSTMs (BiLSTMs) combined with language model features, but found simpler MLPs to be more effective.

For Subtask-2 (user country of origin), they used a simpler classifier due to the challenges of tweet data. They relied on kenlm language model scores for the 21 countries, exponentiated and normalized to sum to 1, and then used a k-Nearest Neighbors (k-NN) classifier. User country was predicted based on the majority prediction across all of a user's tweets.

Datasets

  • Subtask-1: The official training and development data provided by the MADAR organizers, consisting of sentences in different dialects. They also utilized additional data with Arabic sentences in seven dialects.
  • Subtask-2: Tweet-sets downloaded from Twitter, originating from 21 Arabic countries. Participants had to arrange their own tweet downloads due to Twitter's API restrictions.

Key Insights and Ideas

  • Language Models Reign Supreme (for Subtask-1): The paper strongly suggests that language model features, particularly those derived from kenlm, were the most predictive for dialect identification. This highlights the power of capturing sequential patterns in language for distinguishing subtle dialectal differences.
  • Simplicity Trumps Complexity (for Subtask-1): Despite exploring recurrent neural networks (RNNs) and more complex architectures, the authors found that simpler machine learning algorithms (like naive Bayes and SVMs) combined with strong language model features performed better. This is a common theme in NLP: well-engineered features can often outperform complex models on less data.
  • Character-Level Modeling is Key for Informal Text: To adapt language models for character-level analysis, they treated each character as a separate token, effectively creating character n-grams. This allowed them to capture fine-grained patterns that word-level models might miss in informal text.
  • Tweet Data is Messy: Subtask-2 proved significantly more challenging due to the inherent noisiness, incompleteness (unavailable tweets), and informal nature of Twitter data compared to curated corpora. This underscores the difficulty of applying NLP techniques to real-world social media.

Key Results

  • Subtask-1: The Tortuous Classifier achieved a macro F₁-score of 0.658, ranking 6th out of 19 teams. The Neural Network Classifier achieved 0.648.
  • Subtask-2: The system achieved a macro F₁-score of 0.475, ranking 7th out of 9 teams.
  • Error Analysis (Subtask-1): Confusion matrix analysis showed that errors were often geographically plausible, with dialects from nearby cities or regions being confused (e.g., Egyptian dialects being confused with each other, or Levantine dialects being confused).

Limitations and Future Work

  • Subtask-1 Overfitting: The Tortuous Classifier performed better on development data than test data, suggesting potential overfitting.
  • Subtask-2 Data Availability: A significant percentage of tweets were unavailable (around 10-12%), impacting performance. They experimented with a fallback strategy for users with no remaining tweets, which improved accuracy by about 5% but decreased user classification accuracy on the development set.
  • Future Work: The paper implicitly suggests that further exploration of character-level modeling and robust handling of noisy social media data are important directions. The success of language models also points to continued research in this area for dialect identification.

Summary updated

BibTeX
@inproceedings{priban2019zcu,
  author = {Pavel Přibáň and Stephen Taylor},
  title = {{ZCU-NLP at MADAR 2019: Recognizing Arabic Dialects}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure