Abstract
The COVID-19 pandemic has generated a massive volume of discourse on social media, providing a unique window into public health concerns. However, analyzing this data requires specialized tools capable of handling medical terminology and nuanced sentiment within noisy, colloquial text. This paper introduces METS-CoV, a comprehensive dataset of 10,000 COVID-19-related tweets annotated for both Named Entity Recognition (NER) and Targeted Sentiment Analysis (TSA). METS-CoV features seven entity types—including Disease, Drug, Symptom, and Vaccine—and provides sentiment labels for over 9,000 specific entities. Our benchmarks demonstrate that domain-specific pre-training, such as COVID-TWITTER-BERT, is essential for high-performance extraction and analysis in this domain.
Introduction
Public health researchers and epidemiologists increasingly leverage social media to track disease spread and gauge public reaction to health interventions. Despite the potential, social media text presents significant challenges: it is "noisy," characterized by slang, irony, and non-standard grammar. Existing Natural Language Processing (NLP) tools, often trained on formal news corpora (like CoNLL) or scientific literature (like CORD-NER), frequently struggle with the informal nature of Twitter.
Furthermore, standard sentiment analysis often fails to capture the complexity of public opinion. A general model might label a tweet as "neutral" if it contains both positive and negative elements, whereas Targeted Sentiment Analysis (TSA) can pinpoint how a user feels about a specific entity—for instance, expressing frustration with a lockdown while remaining positive about a vaccine. METS-CoV addresses this gap by providing a large-scale, expert-annotated dataset that combines medical entity extraction with targeted sentiment.
The METS-CoV Dataset
The METS-CoV dataset consists of 10,000 tweets selected from an initial pool of over 368 million. The researchers employed a rigorous filtering process, using a symptom-based lexicon to identify 2.2 million potentially relevant medical tweets before sampling the final set.
Annotation and Entities
Unlike many social media datasets that rely on crowdsourced labor, METS-CoV was annotated by individuals with backgrounds in medicine, public health, and pharmacy. This expert-led approach ensured high-quality labels, achieving an 85% F1 inter-annotator agreement for NER.
The dataset supports two primary tasks:
- Named Entity Recognition (NER): Identifying 19,057 entities across seven categories:
- Medical: Disease, Drug, Symptom, Vaccine.
- General: Person, Location, Organization.
- Targeted Sentiment Analysis (TSA): Assigning sentiment (Positive, Negative, Neutral) to 9,101 specific entities (Person, Organization, Drug, and Vaccine) within 5,278 tweets.
Benchmarking and Performance
The authors evaluated several models, ranging from traditional Conditional Random Fields (CRF) to state-of-the-art Transformers.
Domain-Specific Pre-training
A key finding was the superiority of domain-specific models. COVID-TWITTER-BERT consistently outperformed larger, general-purpose models like RoBERTa-large. This suggests that exposure to the specific vocabulary and linguistic "vibe" of pandemic-era Twitter is more critical for performance than raw parameter count. COVID-TWITTER-BERT achieved a micro-F1 score of 83.88 for the NER task.
Targeted Sentiment Results
For TSA, the best-performing architecture utilized a Dependency Graph Convolutional Network (depGCN) with COVID-TWITTER-BERT as a feature extractor. The analysis revealed that neutral sentiments dominate the dataset, reflecting the tendency of users to share news or factual updates. Interestingly, the data showed more positive sentiment toward Drugs than Vaccines, highlighting a potential area for further public health communication research.
Key Insights
- Entity Confusion: Models frequently struggled to distinguish between Symptom and Disease entities, likely because they appear in similar linguistic contexts (e.g., "I am suffering from [X]").
- The Length Penalty: Model performance degrades as tweet length increases beyond 40 tokens. Longer tweets often introduce noise, multiple entities, and complex clauses that challenge current architectures.
- Social Media Nuance: The dataset captures the colloquial nature of medical discussion, making it a more robust benchmark for real-world public health monitoring than clinical or scientific datasets.
Conclusion and Future Work
METS-CoV represents a significant step forward in computational social science and epidemiology. By providing a high-quality, medically-focused dataset, it enables the development of tools that can better understand public discourse during health crises. Future work will focus on addressing entity imbalance—specifically the over-representation of symptoms—and improving model robustness against the inherent subjectivity and irony of social media text.
Summary updated