NeedFull - a Tweet Analysis Platform to Study Human Needs During the COVID-19 Pandemic in New York State

Zijian Long, Rajwa Alharthi, Abdulmotaleb El-SaddikIEEE Access 2020 · OpenReview

TL;DR

This paper introduces NeedFull, a novel tweet analysis platform designed to understand the psychological needs of citizens during crises like the COVID-19 pandemic. By integrating data collection, storage, analysis, and visualization, NeedFull offers a comprehensive view of public sentiment and needs, demonstrated through an analysis of New York State's reaction to the pandemic.

What Problem Does This Paper Address?

In times of crisis, understanding the psychological well-being and needs of the public is crucial for effective governance and support. Traditional methods of gauging public sentiment often focus on surface-level emotions (sentiment analysis) or specific emotions. However, there's a significant gap in tools that can delve deeper into human needs – such as relatedness, competence, and autonomy – and assess their satisfaction levels. This is particularly challenging during rapidly evolving events like pandemics, where public discourse is amplified on social media. The authors aim to bridge this gap by providing a platform that can analyze social media data to identify and understand these deeper psychological needs.

Methods

The core of this work is the NeedFull platform, a four-module system:

  • Data Collection: Utilizes a custom Twitter crawler to gather both recent (past 7 days) and historical tweets, supplementing Twitter's standard API limitations.
  • Data Storage: Employs MongoDB, a NoSQL database, for its scalability and performance with large datasets. It uses a replica set for read operations and sharding for efficient data management.
  • Data Analysis: This is the most novel part, leveraging a human need-detection framework based on psychological theories. This framework involves:
  • Offline Training: Developing models for recognizing need content (NCR), identifying need types (NTI), measuring satisfaction (NSM), evaluating social context (SCE), and identifying life aspects (LAI). This phase uses machine learning algorithms.
  • Online Analysis: Applying these trained models to new, incoming tweets to classify them in real-time.
  • Preprocessing: Includes keeping stop words, stemming, retaining emojis/emoticons, and tokenization.
  • Feature Extraction: Combines text-based features (BoW, N-gram LM), psychological features (LIWC, LCM), and Twitter-specific features (emojis, hashtags).
  • Normalization: Uses Min-Max scaling to ensure features with different ranges contribute equally.
  • Data Visualization: Provides a user-friendly interface with interactive dashboards to present the analysis results, including human needs distribution, life aspect analysis, and social context evaluation.

Datasets

The primary dataset used is publicly accessible English tweets from New York State collected between March 1 and April 5, 2020, totaling 154,486 tweets. These tweets were gathered using hashtags related to COVID-19 (e.g., #coronavirus, #COVID-19, #outbreak, #pandemic, #virus). The paper also references previous work that likely used other Twitter datasets for training the human need detection models, specifically citing [7] and [22].

Key Insights and Ideas

  • Beyond Sentiment: The Importance of Human Needs: The paper strongly advocates for moving beyond simple sentiment analysis to understand deeper psychological needs (relatedness, competence, autonomy) as a more meaningful indicator of public well-being during crises.
  • Social Media as a Crisis Barometer: Twitter data, when analyzed through a psychological lens, can provide real-time, granular insights into how a population is coping with a crisis, revealing shifts in needs and satisfaction levels.
  • The COVID-19 Pandemic's Impact on Needs: The analysis of New York State tweets during early COVID-19 revealed a significant increase in dissatisfied needs, particularly relatedness, as the pandemic progressed. This highlights the profound social and psychological disruption caused by lockdowns and fear.
  • Life Aspects and Social Context Matter: The study shows that social relations, health, and education were the most discussed life aspects, and the social context was often perceived as unsupportive, contributing to overall frustration.
  • Platform Integration is Key: The NeedFull platform's success lies in its integrated approach, seamlessly connecting data collection to actionable insights through visualization.

Key Results

  • The NeedFull platform successfully analyzed 154,486 tweets from New York State during the early COVID-19 pandemic.
  • Need Satisfaction Trends: Weekly analysis showed a general decline in satisfied needs and an increase in frustrated needs throughout the observed period. For instance, satisfied relatedness dropped from 28.35% in Week 1 to 14.43% in Week 5.
  • Dominant Needs: Relatedness was the most pronounced human need discussed (47.42%), followed by autonomy (16.63%) and competence (14.51%).
  • Life Aspect Dominance: Social relation (32%) was the most frequently discussed life aspect, followed by health (18%).
  • Social Context: Unsupportive social contexts were more frequently reported than supportive ones, particularly in later weeks.
  • Model Performance: The human need recognition models, based on SVM, achieved accuracies ranging from 60.48% (LAI) to 93.56% (NSM), as detailed in Table 1.

Limitations and Future Work

  • Text-Only Analysis: The current NeedFull platform is limited to analyzing text content and cannot process image or video data from platforms like Instagram.
  • Future Work: The authors plan to extend the platform to analyze other social media content types, such as images and videos, to provide a more holistic understanding of human needs.

Summary updated

Abstract

Governments and municipalities need to understand their citizens’ psychological needs in critical times and dangerous situations. COVID-19 brings lots of challenges to deal with. We propose NeedFull, an interactive and scalable tweet analysis platform, to help governments and municipalities to understand residents’ real psychological needs during those periods. The platform mainly consists of four parts: data collection module, data storage module, data analysis module and data visualization module. The four parts interact with each other and provide users with a thorough human needs analysis based on their queries. We employed the proposed platform to investigate the reaction of people in New York State to the ongoing worldwide COVID-19 pandemic.
BibTeX
@inproceedings{long2020needfull,
  author = {Zijian Long and Rajwa Alharthi and Abdulmotaleb El-Saddik},
  title = {{NeedFull - a Tweet Analysis Platform to Study Human Needs During the COVID-19 Pandemic in New York State}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=jlOIF58OMA},
}
Figure