TL;DR
This paper introduces a novel, manually annotated corpus of Brazilian Portuguese tweets from political elites regarding COVID-19 vaccines. It distinguishes between stance (opinion on vaccines) and sentiment (emotional tone), addressing a critical gap in resources for this language and topic, and offering a valuable benchmark for NLP research.
What Problem Does This Paper Address?
Analyzing public discourse on critical issues like vaccine policy is crucial, and social media platforms like X (formerly Twitter) are key sources of this data. However, manually processing the sheer volume of user-generated content is impractical. Natural Language Processing (NLP) offers solutions through sentiment analysis (detecting emotional tone) and stance detection (identifying opinions towards a target). While these tasks are well-established for English, there's a significant scarcity of annotated resources in Brazilian Portuguese, especially concerning COVID-19 vaccines. Furthermore, many existing studies conflate sentiment and stance, leading to potential inaccuracies. This paper aims to bridge this gap by creating a high-quality, annotated corpus that meticulously separates these two crucial analytical dimensions.
Methods
The researchers developed a detailed annotation scheme and protocol to classify tweets. This involved:
- Relevance Filtering: Initially, tweets were classified as relevant or irrelevant to the topic of COVID-19 vaccines. This step ensured that subsequent analysis focused on pertinent discussions.
- Stance Detection: Relevant tweets were then categorized into "favorable," "unfavorable," or "unclear" regarding COVID-19 vaccines. The "unclear" category is a novel addition, designed to capture ambiguity or insufficient context, distinguishing it from a neutral stance.
- Sentiment Analysis: Separately, tweets were analyzed for their overall emotional tone, classified as "positive," "negative," or "unclear." This was explicitly decoupled from the stance towards vaccines, acknowledging that a favorable stance doesn't always equate to positive sentiment, and vice-versa.
The annotation process involved nine annotators working in three groups, undergoing extensive training and regular meetings to ensure consistency. Inter-annotator agreement was measured using Krippendorff's alpha, indicating fair to moderate reliability for stance and sentiment.
Datasets
The paper introduces a newly created dataset:
- COVID-19 Vaccine Stance and Sentiment Corpus: This corpus consists of 9,045 tweets in Brazilian Portuguese posted by Brazilian political elites between 2020 and 2022. Of these, 5,937 (65.64%) were deemed relevant to COVID-19 vaccines and subsequently annotated for stance and sentiment. The corpus is made publicly available on GitHub under a Creative Commons license.
This dataset is novel in its focus on Brazilian Portuguese, its specific domain (COVID-19 vaccines), its source (political elites), and its rigorous separation of stance and sentiment.
Key Insights and Ideas
- Stance vs. Sentiment is Crucial: The most significant insight is the critical importance of distinguishing between stance and sentiment. The authors demonstrate with examples that a positive stance can be expressed with negative sentiment (e.g., frustration with government handling of vaccine rollout) and vice-versa. Conflating these can lead to substantial measurement errors.
- "Unclear" as a Valuable Category: The introduction of an "unclear" category for both stance and sentiment is a key methodological contribution. It acknowledges the inherent ambiguity in some online discourse, which is often missed by binary or strictly neutral classifications.
- Political Elites' Discourse Varies: The analysis reveals that while the majority of political elites expressed a favorable stance towards COVID-19 vaccines, the sentiment expressed was more balanced, with a significant portion being negative. This suggests that even supporters of vaccination may express frustration or criticism related to its implementation or broader political contexts.
- Temporal Shifts in Discourse: The availability of vaccines in 2021 marked a notable shift, with a significant increase in relevant posts and a majority expressing positive sentiment, likely reflecting hope and progress.
Key Results
- Annotation Agreement: Krippendorff's alpha scores indicated good agreement for relevance (0.94), moderate for sentiment (0.67), and moderate for stance (0.70).
- Stance Distribution: A strong majority of relevant posts (78.6%) expressed a favorable stance towards COVID-19 vaccines. Unfavorable stances were rare (4.0%), while "unclear" stances constituted a notable portion (17.4%).
- Sentiment Distribution: Sentiment was more balanced, with positive (46.8%) and negative (46.6%) posts being nearly equal, and a smaller percentage of unclear sentiment (6.6%).
- Temporal Trends: In 2021, coinciding with vaccine availability, there was a sharp increase in relevant posts and a shift towards positive sentiment.
- Stance-Sentiment Correlation: While there's an overall association between stance and sentiment, the cross-tabulation shows significant deviations, reinforcing the need for separate classification. For instance, 40.3% of favorable stance posts had negative sentiment.
Limitations and Future Work
Limitations:
- Language Specificity: The corpus is limited to Brazilian Portuguese, which may affect the generalizability of findings to other languages and dialects.
- Timeframe: The analysis covers only 2020-2022, limiting insights into longer-term discourse evolution.
- Manual Annotation Bias: Despite efforts, manual annotation inherently carries potential biases and inconsistencies.
Future Work:
- Expand to Other Languages/Domains: Develop scalable methods for multi-language and multi-domain sentiment and stance analysis.
- Focus on Specific Populations: Explore discourse on COVID-19 vaccines concerning children, adolescents, and vulnerable groups.
- Broader Political Scope: Extend the analysis to national political elites and other X messages within the same domain.
- Machine Learning Applications: The annotated corpus is intended to serve as a gold standard for training and evaluating machine learning models for relevance, sentiment, and stance analysis.
Summary updated