A Deep CNN Architecture with Novel Pooling Layer Applied to Two Sudanese Arabic Sentiment Datasets

Mustafa Mhamed, Richard F. E. Sutcliffe, Xia Sun, Jun Feng, Eiad Almekhlafi, Ephrem A. RettaarXiv 2022 · arXiv · OpenReview

TL;DR

This paper introduces two new datasets for analyzing sentiment in Sudanese Arabic, a widely spoken but under-researched dialect. They propose a novel Convolutional Neural Network (CNN) architecture with a unique "Mean Max Average" (MMA) pooling layer, achieving state-of-the-art results on these new datasets and demonstrating its effectiveness on existing Arabic sentiment analysis benchmarks.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, is a crucial area in natural language processing. While significant research has focused on Modern Standard Arabic (MSA), the formal written form, many real-world Arabic communications occur in various regional dialects. Sudanese Arabic, spoken by 32 million people, has been largely overlooked in sentiment analysis research, lacking dedicated datasets and specialized models. This gap hinders our ability to understand public opinion and sentiment within this significant linguistic community.

Methods

The core of this work lies in two main contributions: new datasets and a novel deep learning architecture.

  • Dataset Creation: The researchers meticulously created two new datasets:
  • SudSenti2: A 2-class (positive/negative) dataset derived from Facebook and YouTube posts.
  • SudSenti3: A 3-class (positive/negative/neutral) dataset sourced from Twitter. Both datasets underwent a rigorous annotation process involving three native Sudanese Arabic speakers to ensure accuracy and dialect specificity.

  • SCM+MMA Architecture: They propose a Sentiment Convolutional Model (SCM) built upon a CNN. The novelty lies in its custom Mean Max Average (MMA) pooling layer. Standard pooling layers like Max pooling (good for local features like edges) and Average pooling (good for global characteristics) have their own strengths and weaknesses. MMA pooling aims to combine the benefits of both by calculating the mean of the max value and the average value within a pooling region. This is intended to capture a richer set of features. The SCM architecture itself consists of an embedding layer, four convolutional layers, the MMA pooling layer, and finally dense and softmax layers for classification.

Datasets

  • SudSenti2: A newly created 2-class dataset (positive/negative) for Sudanese Arabic, collected from Facebook and YouTube. It contains 2,027 positive and 1,973 negative posts, totaling 4,000 entries.
  • SudSenti3: A newly created 3-class dataset (positive/negative/neutral) for Sudanese Arabic, collected from Twitter. It comprises 2,523 positive, 2,639 negative, and 1,947 neutral tweets, totaling 7,109 entries.
  • Saudi Sentiment Dataset (SSD): An existing 2-class dataset for Saudi Arabic, used for comparison.
  • Hotel Arabic Review Dataset (HARD): An existing MSA dataset, also used for comparison.

Key Insights and Ideas

  • Dialectal Nuances Matter: The paper highlights significant differences between Modern Standard Arabic and Sudanese Arabic, not just in vocabulary but also in grammar and meaning, underscoring the need for dialect-specific resources and models.
  • MMA Pooling's Promise: The proposed MMA pooling layer demonstrates a clear advantage over standard Max, Average, and Min pooling techniques. By combining the strengths of max and average operations, it seems to extract more discriminative features for sentiment classification.
  • Data Scarcity is a Bottleneck: The creation of new, high-quality datasets like SudSenti2 and SudSenti3 is crucial for advancing research in under-represented dialects.

Key Results

The proposed SCM+MMA model achieved impressive results across various datasets:

  • SudSenti2 (Sudanese, 2-class): 92.25% accuracy.
  • SudSenti3 (Sudanese, 3-class): 85.23% accuracy.
  • SSD (Saudi, 2-class): 84.02% accuracy.
  • HARD (MSA, 2-class): 88.37% accuracy.

In all cases, the SCM+MMA model outperformed established machine learning (LR, RF, NB, SVM) and deep learning (RNN, CNN, CNN-LSTM) baselines. The MMA pooling layer itself also showed superior performance compared to Max, Avg, and Min pooling when tested within the SCM architecture.

Limitations and Future Work

  • Limitations: While the paper presents strong results, the authors acknowledge that 3-way classification (including neutral) is inherently more challenging due to the ambiguity of neutral sentiments. The study also focuses on specific social media platforms, and the generalizability to other domains might require further investigation.
  • Future Work: The authors plan to incorporate an attention mechanism into their deep learning models. This would allow the model to dynamically focus on the most relevant parts of the text, potentially leading to even better feature extraction and performance, especially when dealing with larger and more complex Arabic text corpora across various dialects.

Summary updated

Abstract

Arabic sentiment analysis has become an important research field in recent years. Initially, work focused on Modern Standard Arabic (MSA), which is the most widely-used form. Since then, work has been carried out on several different dialects, including Egyptian, Levantine and Moroccan. Moreover, a number of datasets have been created to support such work. However, up until now, less work has been carried out on Sudanese Arabic, a dialect which has 32 million speakers. In this paper, two new publicly available datasets are introduced, the 2-Class Sudanese Sentiment Dataset (SudSenti2) and the 3-Class Sudanese Sentiment Dataset (SudSenti3). Furthermore, a CNN architecture, SCM, is proposed, comprising five CNN layers together with a novel pooling layer, MMA, to extract the best features. This SCM+MMA model is applied to SudSenti2 and SudSenti3 with accuracies of 92.75% and 84.39%. Next, the model is compared to other deep learning classifiers and shown to be superior on these new datasets. Finally, the proposed model is applied to the existing Saudi Sentiment Dataset and to the MSA Hotel Arabic Review Dataset with accuracies 85.55% and 90.01%.
BibTeX
@misc{mhamed2022deep,
  author = {Mustafa Mhamed and Richard F. E. Sutcliffe and Xia Sun and Jun Feng and Eiad Almekhlafi and Ephrem A. Retta},
  title = {{A Deep CNN Architecture with Novel Pooling Layer Applied to Two Sudanese Arabic Sentiment Datasets}},
  year = {2022},
  eprint = {2201.12664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.12664},
}
Figure