METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets

Peilin Zhou, Zeqiang Wang, Dading Chong, Zhijiang Guo, Yining Hua, Zichang Su, Zhiyang Teng, Jiageng Wu, Jie Yang

Abstract

The COVID-19 pandemic has generated a massive volume of discourse on social media, providing a unique window into public health concerns. However, analyzing this data requires specialized tools capable of handling medical terminology and nuanced sentiment within noisy, colloquial text. This paper introduces METS-CoV, a comprehensive dataset of 10,000 COVID-19-related tweets annotated for both Named Entity Recognition (NER) and Targeted Sentiment Analysis (TSA). METS-CoV features seven entity types—including Disease, Drug, Symptom, and Vaccine—and provides sentiment labels for over 9,000 specific entities. Our benchmarks demonstrate that domain-specific pre-training, such as COVID-TWITTER-BERT, is essential for high-performance extraction and analysis in this domain.

Introduction

Public health researchers and epidemiologists increasingly leverage social media to track disease spread and gauge public reaction to health interventions. Despite the potential, social media text presents significant challenges: it is "noisy," characterized by slang, irony, and non-standard grammar. Existing Natural Language Processing (NLP) tools, often trained on formal news corpora (like CoNLL) or scientific literature (like CORD-NER), frequently struggle with the informal nature of Twitter.

Furthermore, standard sentiment analysis often fails to capture the complexity of public opinion. A general model might label a tweet as "neutral" if it contains both positive and negative elements, whereas Targeted Sentiment Analysis (TSA) can pinpoint how a user feels about a specific entity—for instance, expressing frustration with a lockdown while remaining positive about a vaccine. METS-CoV addresses this gap by providing a large-scale, expert-annotated dataset that combines medical entity extraction with targeted sentiment.

The METS-CoV Dataset

The METS-CoV dataset consists of 10,000 tweets selected from an initial pool of over 368 million. The researchers employed a rigorous filtering process, using a symptom-based lexicon to identify 2.2 million potentially relevant medical tweets before sampling the final set.

Annotation and Entities

Unlike many social media datasets that rely on crowdsourced labor, METS-CoV was annotated by individuals with backgrounds in medicine, public health, and pharmacy. This expert-led approach ensured high-quality labels, achieving an 85% F1 inter-annotator agreement for NER.

The dataset supports two primary tasks:

  1. Named Entity Recognition (NER): Identifying 19,057 entities across seven categories:
  2. Medical: Disease, Drug, Symptom, Vaccine.
  3. General: Person, Location, Organization.
  4. Targeted Sentiment Analysis (TSA): Assigning sentiment (Positive, Negative, Neutral) to 9,101 specific entities (Person, Organization, Drug, and Vaccine) within 5,278 tweets.

Benchmarking and Performance

The authors evaluated several models, ranging from traditional Conditional Random Fields (CRF) to state-of-the-art Transformers.

Domain-Specific Pre-training

A key finding was the superiority of domain-specific models. COVID-TWITTER-BERT consistently outperformed larger, general-purpose models like RoBERTa-large. This suggests that exposure to the specific vocabulary and linguistic "vibe" of pandemic-era Twitter is more critical for performance than raw parameter count. COVID-TWITTER-BERT achieved a micro-F1 score of 83.88 for the NER task.

Targeted Sentiment Results

For TSA, the best-performing architecture utilized a Dependency Graph Convolutional Network (depGCN) with COVID-TWITTER-BERT as a feature extractor. The analysis revealed that neutral sentiments dominate the dataset, reflecting the tendency of users to share news or factual updates. Interestingly, the data showed more positive sentiment toward Drugs than Vaccines, highlighting a potential area for further public health communication research.

Key Insights

  • Entity Confusion: Models frequently struggled to distinguish between Symptom and Disease entities, likely because they appear in similar linguistic contexts (e.g., "I am suffering from [X]").
  • The Length Penalty: Model performance degrades as tweet length increases beyond 40 tokens. Longer tweets often introduce noise, multiple entities, and complex clauses that challenge current architectures.
  • Social Media Nuance: The dataset captures the colloquial nature of medical discussion, making it a more robust benchmark for real-world public health monitoring than clinical or scientific datasets.

Conclusion and Future Work

METS-CoV represents a significant step forward in computational social science and epidemiology. By providing a high-quality, medically-focused dataset, it enables the development of tools that can better understand public discourse during health crises. Future work will focus on addressing entity imbalance—specifically the over-representation of symptoms—and improving model robustness against the inherent subjectivity and irony of social media text.

Summary updated

Abstract

The COVID-19 pandemic continues to bring up various topics discussed or debated on social media. In order to explore the impact of pandemics on people's lives, it is crucial to understand the public's concerns and attitudes towards pandemic-related entities (e.g., drugs, vaccines) on social media. However, models trained on existing named entity recognition (NER) or targeted sentiment analysis (TSA) datasets have limited ability to understand COVID-19-related social media texts because these datasets are not designed or annotated from a medical perspective. This paper releases METS-CoV, a dataset containing medical entities and targeted sentiments from COVID-19-related tweets. METS-CoV contains 10,000 tweets with 7 types of entities, including 4 medical entity types (Disease, Drug, Symptom, and Vaccine) and 3 general entity types (Person, Location, and Organization). To further investigate tweet users' attitudes toward specific entities, 4 types of entities (Person, Organization, Drug, and Vaccine) are selected and annotated with user sentiments, resulting in a targeted sentiment dataset with 9,101 entities (in 5,278 tweets). To the best of our knowledge, METS-CoV is the first dataset to collect medical entities and corresponding sentiments of COVID-19-related tweets. We benchmark the performance of classical machine learning models and state-of-the-art deep learning models on NER and TSA tasks with extensive experiments. Results show that the dataset has vast room for improvement for both NER and TSA tasks. METS-CoV is an important resource for developing better medical social media tools and facilitating computational social science research, especially in epidemiology. Our data, annotation guidelines, benchmark models, and source code are publicly available (https://github.com/YLab-Open/METS-CoV) to ensure reproducibility.
BibTeX
@inproceedings{zhou2022mets,
  author = {Peilin Zhou and Zeqiang Wang and Dading Chong and Zhijiang Guo and Yining Hua and Zichang Su and Zhiyang Teng and Jiageng Wu and Jie Yang},
  title = {{METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=Qo-Enz4xMb},
}
Figure