Winds of Change: Impact of COVID-19 on Vaccine-Related Opinions of Twitter Users

Soham Poddar, Mainack Mondal, Janardan Misra, Niloy Ganguly, Saptarshi Ghosh

TL;DR

This paper introduces a novel method to automatically track vaccine-related opinions on Twitter, identifying shifts in user stances from pre-COVID to the pandemic era. By analyzing millions of tweets, the researchers uncover key themes driving these changes and highlight the importance of social connections in shaping vaccine attitudes.

What Problem Does This Paper Address?

The COVID-19 pandemic brought vaccine development and adoption to the forefront of global public health discussions. While societal-scale vaccination is crucial for achieving herd immunity, vaccine hesitancy remains a significant challenge. Understanding the nuances of public opinion, particularly how it evolves over time and in response to major events, is vital for effective public health communication and policy. Previous research often relied on surveys or manual analysis, which are time-consuming and may not capture the dynamic nature of online discourse. This paper aims to bridge this gap by developing an automated, large-scale approach to analyze vaccine-related opinions on Twitter, focusing on identifying shifts in user stances and the underlying reasons.

Methods

The core of this research lies in developing a robust pipeline for analyzing Twitter data:

  • Tweet Classification: The authors trained and evaluated several machine learning models, including SVM, FastText, and BERT-based models, to classify individual tweets into "Anti-Vax," "Pro-Vax," or "Neutral" stances. A fine-tuned BERT model, "CT-BERT++," achieved the best performance, reaching a macro F1-score of 0.825 on a combined dataset.
  • User Stance Identification: Using the tweet classifier, they identified the stance of individual users by aggregating their tweet stances over specific time periods. A user was classified if they posted at least three relevant tweets, with a threshold of 70% of those tweets aligning with a particular stance.
  • Topic Modeling: To understand the content of these opinions, they employed a topic modeling approach (based on Bozarth et al., 2020) to identify key themes discussed by Anti-Vaxxers and Pro-Vaxxers. This involved identifying candidate topics, mapping them to coherent themes, and then analyzing their prevalence across different time periods.
  • Stance Change Analysis: The researchers tracked users who exhibited a consistent stance across multiple time periods and identified those who changed their stance (e.g., from Pro-Vax to Anti-Vax, or vice versa). They then analyzed the anti-vax topics discussed by these "stance-changed" users before and after their shift.
  • Social Network Analysis: To explore the influence of social connections, they analyzed the change in the proportion of Pro-Vaxxer and Anti-Vaxxer followers for stance-changed users over time.

Datasets

  • Pre-existing Datasets:
    • Muller dataset: Tweets about vaccines before the COVID-19 pandemic.
  • Cotfas dataset: Tweets about COVID-19 vaccines from November-December 2020.
  • Newly Introduced Dataset: The authors created a new dataset by collecting 15.7 million English tweets from 4 million distinct users spanning January 2018 to March 2021. This dataset was augmented with manually annotated tweets (1,700 tweets) to improve the classification models.

Key Insights and Ideas

  • Dynamic Nature of Opinions: Vaccine opinions are not static. The pandemic significantly influenced public discourse, leading to shifts in user stances.
  • The "Stance-Changed" User: A notable portion of users changed their vaccine-related stance during the pandemic, highlighting the impact of evolving information and personal experiences.
  • Thematic Evolution: While core anti-vax themes like "Health Concerns" persisted, new themes like "Rushed Development" and "Political Influence" became more prominent during the COVID-19 era.
  • Social Influence Matters: The study suggests a correlation between a user's stance and the stances of their social network (followers), indicating that social connections play a role in shaping vaccine attitudes.
  • Targeted Communication is Key: Understanding the specific concerns of different user groups, especially those who have changed their stance, is crucial for developing effective public health messaging.

Key Results

  • High Precision Classifier: The developed CT-BERT++ classifier achieved a precision of 97% in identifying user stances.
  • Significant Discourse Shift: The volume of vaccine-related tweets increased dramatically during the COVID-19 pandemic.
  • Prevalence of Anti-Vax Themes: In the COVID-19 period, themes like "vaccine development being rushed" and "concerns about vaccines being ineffective" became more frequently discussed by Anti-Vaxxers compared to pre-COVID times.
  • Stance Change Statistics:
    • 218 users shifted from Pro-Vaxxer to Anti-Vaxxer between pre-COVID and COVID periods.
  • 329 users shifted from Anti-Vaxxer to Pro-Vaxxer between COVID and COVID-vax periods.
  • Representativeness of Stance-Changed Users: Stance-changed users were found to be representative of the general active Twitter population in terms of social network properties but expressed opinions more frequently.

Limitations and Future Work

  • Bot Detection: While efforts were made to filter out bots, the authors acknowledge that automated bot detection can be prone to false positives, and manual checks were performed on a subset.
  • Data Collection Bias: The use of the Twint library for collecting older tweets resulted in capturing approximately 15.5% of tweets compared to the Twitter API, though analysis suggested minimal bias.
  • Future Work: The authors suggest exploring other forms of Twitter interactions (retweets, replies) to understand stance change, and developing predictive models to identify users likely to change their stance. They also propose designing targeted recommendation algorithms to combat anti-vax echo chambers.

Summary updated

Abstract

Administering COVID-19 vaccines at a societal scale has been deemed as the most appropriate way to defend against the COVID-19 pandemic. This global vaccination drive naturally fueled a possibility of Pro-Vaxxers and Anti-Vaxxers strongly expressing their supports and concerns regarding the vaccines on social media platforms. Understanding this online discourse is crucial for policy makers. This understanding is likely to impact the success of vaccination drives and might even impact the final outcome of our fight against the pandemic. The goal of this work is to improve this understanding using the lens of Twitter-discourse data. We first develop a classifier that categorizes users according to their vaccine-related stance with high precision (97%). Using this method we detect and investigate specific user-groups who posted about vaccines in pre-COVID and COVID times. Specifically, we identify distinct topics that these users talk about, and investigate how vaccine-related discourse has changed between pre-COVID times and COVID times. Finally, for the first time, we investigate the change of vaccine-related stances in Twitter users and shed light on potential reasons for such changes in stance. Our dataset and classifier are available at https://github.com/sohampoddar26/covid-vax-stance.
BibTeX
@inproceedings{poddar2022winds,
  author = {Soham Poddar and Mainack Mondal and Janardan Misra and Niloy Ganguly and Saptarshi Ghosh},
  title = {{Winds of Change: Impact of COVID-19 on Vaccine-Related Opinions of Twitter Users}},
  booktitle = {Icwsm},
  year = {2022},
}
Figure