The SMarT Classifier for Arabic Fine-Grained Dialect Identification

Karima Meftouh, Karima Abidi, Salima Harrat, Kamel SmailiACL 2019

TL;DR

This paper presents the SMarT system, a novel approach to fine-grained Arabic dialect identification that achieved first place in the MADAR shared task. By combining a Multinomial Naïve Bayes classifier with carefully selected word and character n-grams, along with language model probabilities, the system successfully distinguished between dialects from 25 specific cities across the Arab World.

What Problem Does This Paper Address?

Arabic is a language with significant internal diversity, encompassing Modern Standard Arabic (MSA) and numerous regional dialects. Identifying these dialects is crucial for many Natural Language Processing (NLP) applications, such as machine translation and sentiment analysis. While general language identification is relatively mature, distinguishing between closely related dialects, especially in informal text like social media, presents a substantial challenge. The MADAR shared task specifically aimed to address this by focusing on fine-grained dialect identification at the city and country levels, a more granular and difficult problem than previous efforts.

Methods

The SMarT system employs a Multinomial Naïve Bayes classifier. The authors explored various feature sets to optimize performance. Their final model incorporates:

  • Word n-grams: Unigrams and bigrams were found to be most effective, with higher-order n-grams showing performance degradation due to data sparsity.
  • Character n-grams: Ranging from 1 to 5 characters, these features were found to be more robust than word n-grams. The authors experimented with including or excluding spaces between words, finding that including them (character n-grams from 1 to 5, with spaces) yielded the best results.
  • Language Model Probabilities: The inclusion of probabilities from 26 unigram language models further boosted performance.

The authors experimented with other methods like LSTMs and Word2vec, but found them less effective given the limited size of the provided corpora. They utilized the Scikit-learn library in Python for their implementation.

Datasets

The experiments were conducted using data from Subtask 1 of the MADAR shared task, which focuses on the "MADAR Travel Domain Dialect Identification." This dataset is described as being the same as reported in previous work by Bouamor et al. (2018) and Salameh et al. (2018). It consists of two corpora:

  • Corpus-26: A collection of parallel sentences covering dialects from 25 cities and MSA. The training set has 1600 labeled instances per class, and the development set has 200 per class.
  • Corpus-6: Contains 10,000 additional sentences translated into the dialects of five cities (Beirut, Cairo, Doha, Tunis, Rabat) and MSA. This corpus is split into 9,000 training instances and 1,000 development instances per language.

Key Insights and Ideas

  • Simplicity can be powerful: Despite the availability of more complex deep learning models, a well-tuned Naïve Bayes classifier with carefully selected features proved highly effective for this fine-grained task.
  • Character n-grams are crucial for dialect identification: The study highlights that character-level features are more resilient to the variations and noise often found in dialectal text compared to word-level features.
  • The importance of feature engineering: The authors' systematic experimentation with various n-gram combinations and the inclusion of language model probabilities underscore the significant impact of feature selection on classifier performance.
  • Fine-grained dialect identification is a challenging but solvable problem: The success of the SMarT system demonstrates that even with a large number of specific dialect labels (25 cities + MSA), accurate identification is achievable.

Key Results

The SMarT classifier achieved a Macro accuracy of 67.73% and a macro-averaged F1-score of 67.31% on the test set. This performance placed their system first among 19 participants in the MADAR shared task. The paper also shows that the best performing model was the one combining word unigrams and bigrams with character n-grams (1-5, with and without spaces), further enhanced by language model probabilities.

Limitations and Future Work

The authors acknowledge that while their approach is simple and effective, it is ranked first, suggesting that more effort is needed to make it even more powerful and a truly effective tool for the community. The paper does not explicitly detail future work directions beyond this general statement.

Summary updated

BibTeX
@inproceedings{meftouh2019smart,
  author = {Karima Meftouh and Karima Abidi and Salima Harrat and Kamel Smaili},
  title = {{The SMarT Classifier for Arabic Fine-Grained Dialect Identification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure