Public Perceptions around mHealth Applications during COVID-19 Pandemic: A Network and Sentiment Analysis of Tweets in Saudi Arabia

Samar Binkheder, Raniah N. Aldekhyyel, Alanoud AlMogbel, Nora Al-Twairesh, Nuha Alhumaid, Shahad N. Aldekhyyel, Amr JamalInternational Journal of Environmental Research and Public Health 2021 · DOI

TL;DR

This study leverages Twitter data to analyze public perceptions of six Saudi mHealth apps during the COVID-19 pandemic using network and sentiment analysis. It reveals that while most conversations were neutral, "Tawakkalna" and "Tabaud" saw the most engagement, primarily driven by government accounts, highlighting the potential of social media for understanding user experiences and improving health tech.

What Problem Does This Paper Address?

The COVID-19 pandemic spurred the rapid development and deployment of mobile health applications (mHealth apps) globally, including in Saudi Arabia. Understanding how the public perceives and uses these apps is crucial for their effectiveness and for informing future public health initiatives. However, traditional survey methods can be biased and time-consuming. This paper addresses the need for a real-time, large-scale method to gauge public sentiment and interaction patterns surrounding these critical health tools, specifically focusing on six Saudi mHealth apps launched or heavily utilized during the pandemic.

Methods

The researchers employed a two-phase methodology:

  • Network Analysis: They used Twitter's API and NodeXL software to collect tweets related to six specific mHealth apps. This allowed them to construct social networks for each app, mapping user interactions (tweets, retweets, replies, mentions). They analyzed network metrics like the number of users (nodes), conversational relationships (edges), and identified influential accounts using "betweenness centrality" and "PageRank."
  • Sentiment Analysis: A manual annotation process was used to classify tweets into five categories: positive, negative, neutral, indeterminate, and sarcasm. To automate this, they experimented with three machine learning classifiers: Support Vector Machine (SVM) with TF-IDF features, SVM with AraVec word embeddings, and AraBERT. Data augmentation techniques were used to balance the dataset for training.

Datasets

The primary dataset consisted of tweets collected from Twitter using keywords related to six Saudi mHealth apps ("Sehha," "Mawid," "Sehhaty," "Tetamman," "Tawakkalna," and "Tabaud") between April and December 2020. A total of 73,208 tweets were initially retrieved, refined to 5048 for manual annotation. An additional 1847 tweets were collected via data augmentation. The final dataset for sentiment classification comprised 4719 tweets. No new benchmark datasets were introduced; the study utilized existing Twitter data.

Key Insights and Ideas

  • Government-Led Engagement: The most popular apps, "Tawakkalna" and "Tabaud," were characterized by large networks and were heavily influenced by government accounts. This suggests that mandatory use and government endorsement significantly drive public engagement on social media for these apps.
  • Neutral Dominance: The majority of conversations across all apps were neutral, focusing on information seeking, facts, and general inquiries. This indicates Twitter serves as a valuable platform for users to get information and address concerns about mHealth apps.
  • App Overlap Concerns: Negative sentiments often pointed to confusion arising from overlapping features and similar app names, suggesting a need for better integration and clearer app functionalities.
  • Psychological Impact of Notifications: Negative feedback also highlighted the psychological toll of exposure notifications from apps like "Tabaud," raising concerns about anxiety and privacy.
  • AraVec Outperforms: For automated sentiment classification, SVM with AraVec embeddings proved to be the most effective, outperforming both SVM with TF-IDF and the more complex AraBERT model for this specific Arabic tweet dataset.

Key Results

  • Network Size: "Tawakkalna" and "Tabaud" had the largest networks, with "Tawakkalna" having 3076 users and "Tabaud" having 734 users.
  • Conversational Relationships: "Tawakkalna" led in conversational relationships with 5755, followed by "Tabaud" with 1520.
  • Sentiment Distribution: Most conversations were neutral (52.2% in the augmented dataset). "Tetamman" had the highest proportion of positive tweets (35.9%), while "Mawid" and "Sehhaty" had a higher proportion of negative tweets than positive ones.
  • Sentiment Classifier Performance: The best-performing sentiment classifier (SVM with AraVec) achieved an F1-score of 0.85, with an accuracy, precision, and recall of 85%.

Limitations and Future Work

The authors acknowledge several limitations:

  • Twitter API Limits: The number of tweets retrievable via the API was limited.
  • User Demographics: The data primarily reflects younger Twitter users, potentially underrepresenting older demographics.
  • Platform Bias: Focusing solely on Twitter might miss insights from other social media platforms like Instagram and Facebook.
  • Language Bias: The study focused on Arabic tweets, potentially excluding opinions from non-Arabic speakers.

Future work suggested includes:

  • Multi-modal Analysis: Combining social media data with surveys to gain a more comprehensive understanding of user experience, demographics, and challenges.
  • Topic Modeling: Deeper analysis of tweet content to identify specific themes and concerns.
  • Integrating Twitter Metrics: Combining sentiment analysis with metrics like likes, retweets, and followers to assess sentiment interaction rates.
  • Focus on Key Nodes: Analyzing public sentiment by excluding tweets from official accounts and government entities to capture more organic opinions.
  • Broader Social Media Scope: Expanding analysis to other social media platforms and a larger scale of data.

Summary updated

Abstract

A series of mitigation efforts were implemented in response to the COVID-19 pandemic in Saudi Arabia, including the development of mobile health applications (mHealth apps) for the public. Assessing the acceptability of mHealth apps among the public is crucial. This study aimed to use Twitter to understand public perceptions around the use of six Saudi mHealth apps used during COVID-19: "Sehha", "Mawid", "Sehhaty", "Tetamman", "Tawakkalna", and "Tabaud". We used two methodological approaches: network and sentiment analysis. We retrieved Twitter data using specific mHealth apps-related keywords. After including relevant tweets, our final mHealth app networks consisted of a total of 4995 Twitter users and 8666 conversational relationships. The largest networks in size (i.e., the number of users) and volume (i.e., the conversational relationships) among all were "Tawakkalna" followed by "Tabaud", and their conversations were led by diverse governmental accounts. In contrast, the four remaining mHealth networks were mainly led by the health sector and media. Our sentiment analysis approach included five classes and showed that most conversations were neutral, which included facts or information pieces and general inquires. For the automated sentiment classifier, we used Support Vector Machine with AraVec embeddings as it outperformed the other tested classifiers. The sentiment classifier showed an accuracy, precision, recall, and F1-score of 85%. Future studies can use social media and real-time analytics to improve mHealth apps' services and user experience, especially during health crises.
BibTeX
@article{binkheder2021public,
  author = {Samar Binkheder and Raniah N. Aldekhyyel and Alanoud AlMogbel and Nora Al-Twairesh and Nuha Alhumaid and Shahad N. Aldekhyyel and Amr Jamal},
  title = {{Public Perceptions around mHealth Applications during COVID-19 Pandemic: A Network and Sentiment Analysis of Tweets in Saudi Arabia}},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2021},
  doi = {10.3390/ijerph182413388},
}
Figure