Abstract
Social networks allow individuals to communicate and express opinions about products, services, and political affairs. This research introduces a Machine Learning-based (ML-based) classification scheme for analyzing the social network reviews of Yemeni people using data mining techniques. A constructed dataset consisting of 2,000 Modern Standard Arabic (MSA) and Yemeni dialect records was used for training, along with a 300-record test dataset. Four supervised machine learning algorithms were applied: Support Vector Machine (SVM), Naïve Bayes, K-Nearest Neighbors, and Decision Trees. The results show that the SVM algorithm outperformed the others, achieving an accuracy of 90.65% on the training set and 90.00% on the testing set. However, performance was notably influenced by noisy and sarcastic opinions.
Introduction
While sentiment analysis is a mature field for English, Arabic presents unique challenges due to its complex morphology and the vast differences between Modern Standard Arabic (MSA) and regional dialects. Most existing research focuses on MSA or popular dialects like Egyptian or Jordanian, leaving a significant gap in tools tailored for the Yemeni dialect. In the context of social media—where political discourse is often heated and expressed in local vernacular—standard tools often fail. This paper aims to bridge that gap by creating a dedicated dataset and testing a classification pipeline specifically for Yemeni social network reviews.
Methodology
The authors implemented a classic supervised machine learning pipeline with specific adaptations for the Arabic language, as illustrated in the flowchart below.
The methodology consists of several key stages:
- Preprocessing: This involves tokenization, stop-word removal, and "Light Stemming" (removing common prefixes and suffixes without reducing the word to its absolute root).
- Normalization: A manual process was used to unify different forms of letters (e.g., variants of Alif) that are often used interchangeably in informal writing.
- Feature Selection: The researchers used TF-IDF (Term Frequency-Inverse Document Frequency) to weigh word importance and utilized bi-grams to better capture context.
- Classification: Four standard supervised algorithms were compared: SVM, Naïve Bayes (NB), K-Nearest Neighbors (K-NN), and Decision Trees (DT).
Data Collection and Annotation
The study introduced a newly constructed dataset specifically for this task, sourcing data from Twitter and Facebook comments focusing on political discourse.
- Training Set: 2,000 unique records (1,307 negative, 693 positive) consisting of MSA and Yemeni dialects.
- Testing Set: 300 unique records used to validate the model's performance.
- Annotation: The data was manually labeled by two experts and the author to ensure high-quality ground truth.
Key Results and Performance Analysis
Performance was evaluated using Accuracy, Precision, Recall, and F-measure.
- SVM was the clear winner, achieving 90.65% accuracy on the training set and 90.00% on the testing set. It also led in Precision (92.00%) and Recall (85.34%).
- Decision Trees followed with an 87.33% testing accuracy.
- Naïve Bayes was the fastest to train (2:22 minutes) but had the lowest testing accuracy (82.67%).
- K-NN was the slowest (7:18 minutes) due to the computational cost of calculating distances between data points in a large dataset.
Discussion: The Sarcasm Barrier
A significant finding was the identification of sarcasm as a primary source of error. In political reviews, users often use positive words to convey deeply negative meanings (e.g., sarcastically calling a failing state a "superpower"). These nuances are difficult for traditional ML to catch. Furthermore, Yemeni dialects often appear alongside MSA in the same post, necessitating a model that can handle "code-switching" between formal and informal language.
Conclusion and Future Work
While the SVM model showed high accuracy, the authors acknowledge limitations regarding class imbalance (more negative than positive reviews) and the difficulty of sarcasm detection. Future work aims to incorporate specific sarcasm detection modules, expand the dataset beyond 2,300 records, and explore Deep Learning architectures to capture more complex linguistic patterns.
Summary updated