TL;DR
This paper analyzes Saudi Arabian Twitter sentiment towards distance learning during the COVID-19 pandemic using machine learning. They developed a robust sentiment analysis model that achieved 89.9% accuracy, revealing generally positive sentiment for K-12 education but negative sentiment for university-level learning, with specific issues like platform glitches and exam policies being key drivers.
What Problem Does This Paper Address?
The COVID-19 pandemic necessitated a rapid shift to distance learning globally, presenting significant challenges for educational systems, students, and parents. While the impact of this transition is widely felt, understanding public sentiment, particularly in non-English speaking regions, is crucial for improving educational strategies. This research specifically addresses the gap in understanding the sentiment of Arabic-speaking users in Saudi Arabia regarding distance learning during this unprecedented period. The authors aim to quantify this sentiment and identify the underlying reasons, providing actionable insights for the Saudi Ministry of Education.
Methods
The researchers employed a multi-stage approach for sentiment analysis:
- Data Collection: They collected a large dataset of Arabic tweets related to distance learning in Saudi Arabia using the Twint Python library, focusing on keywords and hashtags. Two datasets were created: an initial dataset for training and testing models, and a second "application" dataset for real-world sentiment analysis.
- Pre-processing: A comprehensive suite of pre-processing steps was applied to clean the tweets, including removing irrelevant content, non-Arabic characters, symbols, URLs, user mentions, punctuation, and stop words. They also performed Arabic text normalization and stemming to standardize the language.
- Feature Extraction: Two common techniques were used: N-gram (unigram and bigram) and Term Frequency-Inverse Document Frequency (TF-IDF). These methods convert textual data into numerical representations that machine learning models can process.
- Classification Models: Six machine learning algorithms were evaluated: Logistic Regression (LR), Naïve Bayes (NB), K-Nearest Neighbors (KNN), XGBoost (XGB), Support Vector Machine (SVM), and Random Forest (RF).
- Model Selection: The models were trained and tested on the manually labeled dataset. Performance was measured using accuracy, precision, recall, and F1-score. The model with the highest accuracy was selected for further analysis.
- Sentiment Analysis on Application Dataset: The best-performing model was then applied to the unlabeled "application" dataset to analyze sentiment across different educational stages (primary/kindergarten, intermediate/high school, university) and identify common themes.
Datasets
- Initial Dataset: Collected from March 2020 to August 2020, consisting of approximately 70,000 Arabic tweets. This dataset was manually labeled as positive or negative for training and testing the sentiment analysis models.
- Application Dataset: Collected during the academic year 2020/2021 (August 2020 to April 2021), comprising around 92,000 Arabic tweets. This dataset was used to apply the best-performing sentiment analysis model and analyze sentiment across educational stages.
Key Insights and Ideas
- The Power of Localized Sentiment Analysis: This study highlights the importance of conducting sentiment analysis in the native language and context of the users. Relying solely on English-language data would miss crucial insights into the experiences of specific populations like those in Saudi Arabia.
- Divergent University vs. K-12 Sentiment: The most striking finding is the stark contrast in sentiment between university students and younger learners. While K-12 students generally expressed positive views, university students showed significant negativity. This suggests that the challenges and impacts of distance learning are not uniform across all educational levels.
- Exam Policies as a Major Driver of Negative Sentiment: For university students, the decision to hold final exams in-person, despite the pandemic, was a primary source of negative sentiment. This points to concerns about safety and a desire for online exam options.
- Platform Glitches and School Rules Matter: For unspecified educational stages, negative sentiment was often linked to technical issues like Blackboard glitches and dissatisfaction with school rules related to distance learning. This indicates that the user experience with the technology and the clarity of guidelines are critical factors.
- Madrasati Platform as a Positive Enabler: The Madrasati Platform, developed by the Ministry of Education, emerged as a key positive factor, particularly for general education stages. Its role in facilitating the continuation of education during the pandemic was well-received.
Key Results
- Model Accuracy: The Logistic Regression classifier with unigram and TF-IDF feature extraction achieved the highest accuracy of 0.899 on the balanced dataset.
- Overall Sentiment: The analysis of the application dataset revealed generally positive opinions regarding distance learning for primary, intermediate, and high school stages.
- University Sentiment: University students expressed predominantly negative opinions about distance learning.
- Top Negative Themes (University): Final exams, in-person exam policies, and requests for online exams were the most frequent concerns.
- Top Negative Themes (Unspecified): Blackboard glitching, platform issues, and school rules were prominent.
- Top Positive Themes (K-12): The Ministry of Education and the Madrasati Platform were frequently mentioned positively.
Limitations and Future Work
- Limitations:
- The study acknowledges that the Arabic language, especially Saudi dialects, presents unique challenges for sentiment analysis due to spelling inconsistencies and lack of capitalization.
- The dataset was collected within Saudi Arabia, limiting the generalizability of findings to other regions or countries.
- The study focused on sentiment related to distance learning; other COVID-19 related issues were not explored.
- Future Work:
- Collect data to study sentiment regarding other COVID-19 related issues, such as vaccine perceptions.
- Investigate and apply more advanced pre-processing techniques, including frameworks for handling Arabic negation.
- Conduct comparative performance analysis using deep learning models and transformer-based language representations (e.g., BERT) for Arabic.
- Expand the study's geographical scope to include different areas and countries.
- Investigate the effect of other COVID-19 prevention measures on public sentiment.
Summary updated