LIUM-MIRACL Participation in the MADAR Arabic Dialect Identification Shared Task

Saméh Kchaou, Fethi Bougares, Lamia Hadrich-BelguithACL 2019

TL;DR

This paper details the LIUM-MIRACL team's participation in the MADAR Arabic Dialect Identification Shared Task, focusing on identifying dialects within the travel domain. They explored various machine learning and deep learning approaches, ultimately achieving a respectable F1-score of 65.41% on the challenging task of distinguishing between 25 Arabic dialects using only the provided training data.

What Problem Does This Paper Address?

Identifying the specific dialect of an Arabic speaker is a crucial but complex task for many Natural Language Processing (NLP) applications, including machine translation and speech recognition. Arabic, unlike many languages, has a vast number of spoken dialects that often lack standardized written forms, making dialect identification a significant challenge. Previous work has typically focused on a smaller number of dialect groups (e.g., 5 main regions). The MADAR Shared Task aimed to push the boundaries by addressing a much larger set of 25 distinct Arabic dialects, making the discrimination task considerably harder due to increased class dissimilarity. This paper tackles the problem of building effective dialect identification systems for this expanded set of dialects, specifically within the travel domain.

Methods

The LIUM-MIRACL team experimented with two main categories of methods:

  • Conventional Machine Learning: They employed Multinomial Naive Bayes (MNB) classifiers. These were trained using various combinations of n-gram features, including word n-grams and character n-grams, with different ranges (1-grams to 5-grams). Term Frequency-Inverse Document Frequency (TF-IDF) was used to score these features. They also explored the impact of data pre-processing, comparing systems trained on raw data versus pre-processed data (which involved normalization of Arabic characters, deletion of short vowels, and punctuation removal).
  • Deep Learning: While mentioned as explored, the paper notes that deep learning approaches did not perform well on this specific task. They investigated "various deep learning sequence2sequence architectures" but do not elaborate on the specifics or reasons for their underperformance in this paper.
  • Language Models (LMs) as Features: They integrated language model scores as additional features into the MNB classifier. This involved training 26 language models (one for each dialect) using word and character sequences, both forward and backward, and then using their perplexity scores as input to the MNB.

Datasets

The primary dataset used was the MADAR corpus, specifically for Subtask 1: Travel Domain Dialect Identification. This corpus was created by translating sentences from the Basic Traveling Expression Corpus (BTEC). The authors utilized two versions:

  • Corpus-26: A smaller-scale parallel corpus containing 25 dialects plus Modern Standard Arabic (MSA). This was used for training and development.
  • Corpus-6: A larger-scale corpus with 5 regional representative dialects and MSA.

The official evaluation was performed on the Corpus-26-Test set. The authors note that they only used the provided training and development data and were constrained from using external manually annotated datasets.

Key Insights and Ideas

  • Raw Data Outperforms Pre-processing: Surprisingly, the MNB systems trained on raw, un-preprocessed data achieved better dialect identification results than those trained on cleaned data. The authors hypothesize this might be due to the translation process, where a single translator per dialect might introduce stylistic variations (spelling, punctuation) that the system learns to distinguish.
  • Deep Learning Struggles: Despite the general success of deep learning in NLP, it did not yield good results for this specific dialect identification task. This suggests that the nuances of dialectal variation might be better captured by traditional n-gram features or that the deep learning architectures explored were not optimally suited.
  • Close Dialects are the Toughest: The analysis of the confusion matrix revealed that dialects from geographically close regions, particularly TUN (Tunisian) and SFX (Sfaxian), were the most difficult to distinguish. This is attributed to their high lexical similarity and the difficulty even native speakers had in differentiating them.

Key Results

  • The best performing system, a Multinomial Naive Bayes classifier using word and character n-grams on raw data, achieved an F1-score of 65.41% on the official Corpus-26-Test set.
  • The best MNB system using word n-grams and character n-grams (1-gram word, 1-5 gram character) on raw data achieved an F1-score of 65.41% on the test set.
  • Using language model scores as features for the MNB classifier decreased the F1-score, indicating that this integration was not beneficial for their system.
  • The TUN vs. SFX confusion matrix showed that out of 200 test sentences for each dialect, 18 SFX were misclassified as TUN, and 41 TUN were misclassified as SFX.

Limitations and Future Work

  • Deep Learning Underperformance: The paper acknowledges that deep learning approaches did not perform well, which is a significant limitation and an area for future exploration to understand why.
  • Raw Data Anomaly: The finding that raw data performed better than pre-processed data is counter-intuitive and warrants further investigation into the nature of the translation process and its impact on feature learning.
  • Effective Limit of Dialect Identification: The authors conclude that the task is reaching its effective limit when distinguishing between dialects with high lexical overlap, as exemplified by the TUN-SFX confusion. Future work could focus on more sophisticated feature engineering or exploring methods that can better capture subtle dialectal differences.

Summary updated

BibTeX
@inproceedings{kchaou2019lium,
  author = {Saméh Kchaou and Fethi Bougares and Lamia Hadrich-Belguith},
  title = {{LIUM-MIRACL Participation in the MADAR Arabic Dialect Identification Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure