Neural Network Architectures for Arabic Dialect Identification

Elise Michon, Minh Quang Pham, Josep Crego, Jean SenellartCOLING 2018

TL;DR

This paper explores various neural network architectures for identifying Arabic dialects from speech. The authors found that a Multi-Input Convolutional Neural Network (CNN) that combines lexical, phonetic, and acoustic features achieved the best performance, outperforming traditional SVMs and other neural network designs. This work highlights the effectiveness of leveraging multiple data modalities for dialect identification, even with limited training data.

What Problem Does This Paper Address?

Dialect identification (DID) is the task of automatically determining the dialect of a spoken or written utterance. This is a challenging problem because dialects of the same language, like Arabic, are closely related, sharing significant lexical and grammatical similarities. Accurate dialect identification is crucial for various Natural Language Processing (NLP) tasks, such as improving machine translation and automatic speech recognition systems, making them more tailored to specific regional variations. The Arabic Dialect Identification (ADI) subtask, specifically, aims to distinguish between Modern Standard Arabic (MSA) and several major Arabic dialects (Egyptian, Gulf, Levantine, and North African). A key challenge in this field, particularly for neural network approaches, is the limited amount of labeled training data available for each dialect.

Methods

The authors experimented with three main neural network architectures:

  • Multi-Input CNN (Run 1): This architecture is designed to learn independent embeddings for different input types (lexical, phonetic, and acoustic features) and then combine them. It uses CNNs with varying filter sizes to process character-level and phone-level transcriptions, concatenates these with acoustic embeddings, and passes them through fully-connected layers for classification. This approach aims to leverage the strengths of each data modality.
  • CNN-biLSTM (Run 2): This is an end-to-end model that takes raw audio features (log-amplitude mel-spectrograms) as input. It uses CNN layers to capture local spatial features within the audio signal and then employs a bidirectional Long Short-Term Memory (biLSTM) layer to model sequential dependencies. This architecture aims to learn acoustic representations directly from the speech data.
  • Binary Classification with CNN-biLSTM (Run 3): This approach adapts the CNN-biLSTM architecture by training five separate binary classifiers, each designed to distinguish one dialect from all others. The final prediction is made by selecting the dialect for which the corresponding binary classifier outputs the highest probability. This strategy aims to simplify the classification problem by breaking it down into smaller, more manageable binary tasks.

Datasets

The primary dataset used in this work is the Arabic Dialect Identification (ADI) dataset from the DSL (Discriminating between Similar Languages, Varieties and Dialects) shared task. This dataset includes:

  • Word transcripts: Provided in Buckwalter format, generated by an Automatic Speech Recognition (ASR) system.
  • Phonetic features: Transcripts from four phoneme recognizers.
  • Acoustic features:
    • i-vectors: Previously used features.
  • Mel-Frequency Cepstral Coefficients (MFCCs) / Spectrograms: Used in the end-to-end CNN-biLSTM models.
  • Acoustic embeddings: Extracted from a pre-trained CNN system.

The dataset is split into training, development (dev), and testing sets. Two test sets were used: MGB-3 (from multi-domain YouTube videos) and a "YouTube surprise" test set. The paper notes inconsistencies in dialect labeling and the presence of duplicate audio files in the training set.

Key Insights and Ideas

  • Multi-modality is key: The best-performing system (Multi-Input CNN) demonstrated that combining lexical, phonetic, and acoustic features yields superior results compared to using any single feature type or even combining only two. This suggests that different modalities capture complementary information for dialect identification.
  • Acoustic features are highly informative: Both SVM and CNN experiments showed that acoustic features alone often perform as well as or better than combinations of features, underscoring their importance for distinguishing Arabic dialects, especially given the phonetic variations.
  • End-to-end models show promise but need refinement: While the CNN-biLSTM models operating directly on audio data were less competitive than the Multi-Input CNN, they represent a move towards more end-to-end approaches. The binary classification strategy with CNN-biLSTMs showed improvement over the multi-class version, suggesting that simplifying the task can be beneficial.
  • Data limitations are a significant hurdle: The authors repeatedly mention that the limited size of the training dataset is a major handicap for neural network models, which are typically data-hungry. This limitation likely explains why their end-to-end models did not outperform the feature-rich Multi-Input CNN.
  • Code-switching and MSA dominance pose challenges: The presence of code-switching between dialects and MSA, along with the dominance of MSA in transcriptions, complicates dialect identification. The confusion matrix for the CNN-biLSTM models revealed particular difficulty in distinguishing Levantine and MSA, possibly due to their typological closeness and the prevalence of MSA in their audio files.

Key Results

  • Best performing system: The Multi-Input CNN (run 1) achieved a macro-averaged F1 score of 0.5289, ranking 3rd in the competition.
  • SVM baseline: A multi-class SVM achieved an F1 score of 0.5270. Notably, when using the new acoustic embeddings trained on train and dev sets, SVM performance improved to 0.5905.
  • CNN-biLSTM (run 2): This end-to-end model achieved an F1 score of 0.3894 at submission time, later improved to 0.4235 with different filter configurations.
  • Binary CNN-biLSTMs (run 3): This approach yielded an F1 score of 0.4339.
  • Performance on test sets: All systems performed better on the MGB-3 test set compared to the YouTube test set, likely due to MGB-3's smaller size.
  • Error analysis: The confusion matrix for the best system showed that Levantine utterances were often misclassified as North African, despite these dialects not being geographically or typologically close.

Limitations and Future Work

  • Limited training data: The primary limitation acknowledged is the insufficient size of the training dataset for neural networks, hindering their ability to learn robust representations and potentially leading to overfitting.
  • Data quality issues: Inconsistencies in dialect labeling, duplicate audio files, and the scarcity and inaccuracy of word transcripts (especially for dialectal speech) pose significant challenges.
  • Future Directions:
    • The authors encourage further research with neural networks in future DSL shared tasks, suggesting that the field is still evolving.
  • They advocate for a dedicated task for written Arabic Dialect Identification, similar to the MGB challenge for spoken ADI, given that informal text is a common data type.
  • The paper suggests exploring how dialect-specific or task-specific features can be shifted towards simpler, more powerful architectures that learn useful representations for Dialect Identification while potentially optimizing other tasks like ASR.

Summary updated

BibTeX
@inproceedings{michon2018neural,
  author = {Elise Michon and Minh Quang Pham and Josep Crego and Jean Senellart},
  title = {{Neural Network Architectures for Arabic Dialect Identification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure