Abstract
This paper presents a dialect identification (DID) system based on the transformer neural network architecture. The conventional convolutional neural network (CNN)-based systems use the shorter receptive fields. We believe that long range information is equally important for language and DID, and self-attention mechanism in transformer captures the long range dependencies. In addition, to reduce the computational complexity, self-attention with downsampling is used to process the acoustic features. This process extracts sparse, yet informative features. Our experimental results show that transformer outperforms CNN-based networks on the Arabic dialect identification (ADI) dataset. We also report that the score-level fusion of CNN and transformer-based systems obtains an overall accuracy of 86.29% on the ADI17 database.
Introduction
Dialect identification (DID) is a complex task within speech processing, often more difficult than general language identification. Dialects within a single language family, such as Arabic, share significant acoustic, linguistic, and speaker characteristics. Arabic is particularly challenging due to its numerous regional dialects, many of which are mutually unintelligible, and the lack of a unified orthographic standard for transcript-based approaches.
While recent end-to-end DID systems have predominantly utilized Convolutional Neural Networks (CNNs) or ResNets, these architectures possess limited receptive fields. This limitation hinders their ability to capture long-range temporal and phonotactic information—cues essential for distinguishing dialects over extended speech sequences. This research adapts the Transformer architecture, leveraging its self-attention mechanism to model these long-range dependencies effectively.
Methodology
The proposed system utilizes the encoder portion of a Speech-Transformer architecture.
Feature Extraction and Downsampling
The model processes 80-dimensional Fbank acoustic features. A significant challenge in applying Transformers to speech is the length of audio sequences, which leads to quadratic complexity ($O(n^2)$) in self-attention. To mitigate this, the authors implement a downsampling and stacking strategy: four adjacent frames are stacked and then downsampled by a factor of three. This not only reduces computational load but also improves accuracy by approximately 7% by forcing the model to focus on sparser, more informative acoustic transitions.
Architecture
The core architecture consists of four Transformer encoder layers, each featuring eight attention heads. Standard sinusoidal positional encodings are added to the downsampled features to provide sequence order information. The output of these layers undergoes global average pooling before passing through fully connected layers for final classification into one of 17 dialects.
Experimental Setup
The system was evaluated using the ADI17 (Arabic Dialect Identification-17) dataset from the MGB-5 challenge. The dataset comprises approximately 3,000 hours of training data sourced from YouTube, covering 17 dialects. Notably, the dataset is highly imbalanced; for instance, the Iraqi (IRA) dialect has over 291,000 utterances, whereas the Jordanian (JOR) dialect has only 5,500. The test set is categorized by utterance duration: short (<5s), medium (5-20s), and long (>20s).
Results and Discussion
The Transformer model demonstrated superior performance over the CNN baseline, particularly on longer speech segments.
Performance Comparison
The standalone Transformer achieved an overall test accuracy of 82.54%, compared to 77.77% for the CNN. As hypothesized, the Transformer's advantage grew with utterance length, reaching 90.58% accuracy on segments longer than 20 seconds.
Model Fusion
A score-level fusion, which averages the output probabilities of the CNN and Transformer models, yielded the best results with an overall accuracy of 86.29%. This suggests that the local feature extraction of CNNs and the global context modeling of Transformers are highly complementary.
Data Imbalance
Performance was strongly correlated with the volume of training data. The Iraqi dialect achieved 96.2% accuracy, while the data-scarce Jordanian dialect reached only 68.5%, frequently being confused with geographically proximate dialects like Palestinian and Lebanese.
Conclusion
This work demonstrates that Transformers are highly effective for Arabic Dialect Identification, especially when combined with downsampling to manage long sequences. While the Transformer excels at capturing long-range context, it slightly underperforms on very short clips where downsampling may remove critical information. Future work will investigate more sophisticated positional encodings and the integration of multi-modal features, such as ASR transcripts and lexical data, to further enhance identification accuracy.
Summary updated