TL;DR
This study leverages Twitter data to analyze public perceptions of six Saudi mHealth apps during the COVID-19 pandemic using network and sentiment analysis. It reveals that while most conversations were neutral, "Tawakkalna" and "Tabaud" saw the most engagement, primarily driven by government accounts, highlighting the potential of social media for understanding user experiences and improving health tech.
What Problem Does This Paper Address?
The COVID-19 pandemic spurred the rapid development and deployment of mobile health applications (mHealth apps) globally, including in Saudi Arabia. Understanding how the public perceives and uses these apps is crucial for their effectiveness and for informing future public health initiatives. However, traditional survey methods can be biased and time-consuming. This paper addresses the need for a real-time, large-scale method to gauge public sentiment and interaction patterns surrounding these critical health tools, specifically focusing on six Saudi mHealth apps launched or heavily utilized during the pandemic.
Methods
The researchers employed a two-phase methodology:
- Network Analysis: They used Twitter's API and NodeXL software to collect tweets related to six specific mHealth apps. This allowed them to construct social networks for each app, mapping user interactions (tweets, retweets, replies, mentions). They analyzed network metrics like the number of users (nodes), conversational relationships (edges), and identified influential accounts using "betweenness centrality" and "PageRank."
- Sentiment Analysis: A manual annotation process was used to classify tweets into five categories: positive, negative, neutral, indeterminate, and sarcasm. To automate this, they experimented with three machine learning classifiers: Support Vector Machine (SVM) with TF-IDF features, SVM with AraVec word embeddings, and AraBERT. Data augmentation techniques were used to balance the dataset for training.
Datasets
The primary dataset consisted of tweets collected from Twitter using keywords related to six Saudi mHealth apps ("Sehha," "Mawid," "Sehhaty," "Tetamman," "Tawakkalna," and "Tabaud") between April and December 2020. A total of 73,208 tweets were initially retrieved, refined to 5048 for manual annotation. An additional 1847 tweets were collected via data augmentation. The final dataset for sentiment classification comprised 4719 tweets. No new benchmark datasets were introduced; the study utilized existing Twitter data.
Key Insights and Ideas
- Government-Led Engagement: The most popular apps, "Tawakkalna" and "Tabaud," were characterized by large networks and were heavily influenced by government accounts. This suggests that mandatory use and government endorsement significantly drive public engagement on social media for these apps.
- Neutral Dominance: The majority of conversations across all apps were neutral, focusing on information seeking, facts, and general inquiries. This indicates Twitter serves as a valuable platform for users to get information and address concerns about mHealth apps.
- App Overlap Concerns: Negative sentiments often pointed to confusion arising from overlapping features and similar app names, suggesting a need for better integration and clearer app functionalities.
- Psychological Impact of Notifications: Negative feedback also highlighted the psychological toll of exposure notifications from apps like "Tabaud," raising concerns about anxiety and privacy.
- AraVec Outperforms: For automated sentiment classification, SVM with AraVec embeddings proved to be the most effective, outperforming both SVM with TF-IDF and the more complex AraBERT model for this specific Arabic tweet dataset.
Key Results
- Network Size: "Tawakkalna" and "Tabaud" had the largest networks, with "Tawakkalna" having 3076 users and "Tabaud" having 734 users.
- Conversational Relationships: "Tawakkalna" led in conversational relationships with 5755, followed by "Tabaud" with 1520.
- Sentiment Distribution: Most conversations were neutral (52.2% in the augmented dataset). "Tetamman" had the highest proportion of positive tweets (35.9%), while "Mawid" and "Sehhaty" had a higher proportion of negative tweets than positive ones.
- Sentiment Classifier Performance: The best-performing sentiment classifier (SVM with AraVec) achieved an F1-score of 0.85, with an accuracy, precision, and recall of 85%.
Limitations and Future Work
The authors acknowledge several limitations:
- Twitter API Limits: The number of tweets retrievable via the API was limited.
- User Demographics: The data primarily reflects younger Twitter users, potentially underrepresenting older demographics.
- Platform Bias: Focusing solely on Twitter might miss insights from other social media platforms like Instagram and Facebook.
- Language Bias: The study focused on Arabic tweets, potentially excluding opinions from non-Arabic speakers.
Future work suggested includes:
- Multi-modal Analysis: Combining social media data with surveys to gain a more comprehensive understanding of user experience, demographics, and challenges.
- Topic Modeling: Deeper analysis of tweet content to identify specific themes and concerns.
- Integrating Twitter Metrics: Combining sentiment analysis with metrics like likes, retweets, and followers to assess sentiment interaction rates.
- Focus on Key Nodes: Analyzing public sentiment by excluding tweets from official accounts and government entities to capture more organic opinions.
- Broader Social Media Scope: Expanding analysis to other social media platforms and a larger scale of data.
Summary updated