Introduction
The COVID-19 pandemic catalyzed a global shift in public discourse toward social media platforms, where debates over health mandates—such as masking and school closures—became highly polarized. Understanding this discourse requires more than just identifying whether a user supports or opposes a mandate; it requires identifying the underlying reasoning, or "premise," behind their stance. This paper introduces a new manually annotated dataset of over 6,000 English tweets designed to advance argument mining in the public health domain.
The Challenge of Argument Mining in Public Health
While Natural Language Processing (NLP) has made significant strides in stance detection (identifying if a user is "pro" or "anti" a topic), premise classification remains a challenging frontier. A stance expresses a position, whereas a premise provides the "why." For instance, a tweet stating "Masks are bad!" expresses a stance, but "Masks are bad because they restrict oxygen" includes a premise. Distinguishing between purely emotional claims and reasoned arguments is essential for health officials to understand public concerns and tailor communication strategies effectively.
Methodology and Data Curation
The authors developed this dataset for "Task 2" of the SMM4H 2022 workshop. The process involved:
- Data Collection: Leveraging and expanding an existing stance dataset with premise labels.
- Annotation: Utilizing crowdsourcing via Yandex.Toloka with three annotators per tweet. Quality was maintained through "control tasks" to filter unreliable workers.
- Baseline Establishment: Benchmarking performance using a Random baseline and a fine-tuned RoBERTa-base model.
The resulting corpus consists of 6,156 English tweets covering three controversial topics: school closures, stay-at-home orders, and face masks. The training set includes 3,556 tweets, while the validation and test sets contain 600 and 2,000 tweets, respectively.
Key Insights
Several critical findings emerged from the dataset analysis and the subsequent shared task:
- Task Difficulty: Stance detection (a three-way classification: favor, against, neither) proved significantly harder for models than the binary task of premise identification.
- Contextual Importance: Top-performing systems benefited from appending the claim text (e.g., "wearing a face mask") to the tweet, treating the problem as a "Next Sentence Prediction" task to provide necessary context.
- Sentiment vs. Argument: The authors highlight that strong sentiment does not necessarily imply a reasoned argument. Separating these elements is key to building nuanced monitoring tools.
Results and Performance
The RoBERTa-base model established a strong baseline, particularly in premise classification. While competition participants significantly improved upon the stance detection baseline, the premise detection baseline remained highly competitive.
On the test set, the RoBERTa baseline achieved an $F_1$ score of 0.45 for stance detection and 0.722 for premise classification. The best participant system reached an $F_1$ of 0.64 for stance, highlighting the potential for advanced BERT-based architectures to handle complex social media text. Interestingly, the best system for premise detection reached an $F_1$ of 0.70, which was slightly lower than the authors' own RoBERTa baseline.
Limitations and Future Directions
The current dataset is limited to three COVID-19 mandates in English. Future work aims to expand this to a broader range of health topics, such as vaccination and alternative treatments, and to explore multilingual datasets. Additionally, the authors plan to investigate multi-task learning architectures to improve the simultaneous detection of stance and premise.
Summary updated