METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets

Peilin Zhou, Zeqiang Wang, Dading Chong, Zhijiang Guo, Yining Hua, Zichang Su, Zhiyang Teng, Jiageng Wu, Jie Yang

TL;DR

Researchers have released METS-CoV, the first comprehensive dataset specifically designed to track medical entities and targeted sentiments in COVID-19 tweets. By annotating 10,000 tweets with medical-specific labels and user attitudes toward drugs and vaccines, the authors provide a much-needed benchmark for building NLP tools that can help epidemiologists monitor public health concerns in real-time.

What Problem Does This Paper Address?

During a global health crisis, social media is a goldmine for understanding public sentiment, but mining that gold is notoriously difficult. Standard Natural Language Processing (NLP) tools often stumble when applied to COVID-19 tweets for two main reasons:

  1. Domain Mismatch: Most Named Entity Recognition (NER) models are trained on news articles or general social media. They aren't "medical-literate" enough to distinguish between a casual mention of a symptom and a clinical discussion of a drug.
  2. Noisy Data: Social media text is messy, filled with slang, irony, and abbreviations that trip up models designed for formal scientific papers.

While datasets like CORD-NER exist for scientific literature, there was a glaring gap for a dataset that looks at medical entities through the lens of public health on social media. Without this, we cannot accurately track how people feel about specific vaccines or which side effects are being discussed most frequently.

Methods and Dataset Construction

The authors built METS-CoV using a rigorous, multi-stage pipeline starting with a massive pool of over 368 million tweets. They filtered for English-only content and used a symptom-based lexicon to narrow the focus to medical-related discussions.

Unlike many datasets that rely on general crowdsourcing, the annotators for METS-CoV came from medical, public health, and pharmaceutical backgrounds. They followed a strict three-phase process (pre-annotation training, formal labeling in pairs, and final quality control) to ensure high inter-annotator agreement.

Example annotations in METS-CoV showing entities and targeted sentiments

Instead of just labeling a tweet as "positive" or "negative," the team performed Targeted Sentiment Analysis (TSA). This means if a tweet mentions both "Governor Cuomo" and a "malaria drug," the model must identify the sentiment toward each entity separately.

The primary contribution is the METS-CoV dataset itself:

  • Size: 10,000 tweets.
  • NER Labels: 7 entity types (4 medical: Disease, Drug, Symptom, Vaccine; 3 general: Person, Location, Organization).
  • TSA Labels: 9,101 entities (within 5,278 tweets) annotated with Positive, Negative, or Neutral sentiment.

Key Insights and Results

The paper benchmarks METS-CoV against general and medical datasets, testing a wide array of models ranging from traditional CRFs to state-of-the-art Transformers like BERT, RoBERTa, and domain-specific variants like COVID-TWITTER-BERT.

Model performance by tweet length and confusion matrix of entity predictions
  • Domain-Specific Pre-training is King: The clear superiority of COVID-TWITTER-BERT was a major takeaway. It achieved a micro-F1 score of 83.88 on the NER task, significantly outperforming general-purpose models like RoBERTa-large. This proves that for niche tasks, training on the specific "dialect" of the domain (COVID-related Twitter-speak) is more important than just having a larger model.
  • The "Symptom vs. Disease" Confusion: The confusion matrix revealed that models frequently mix up Symptom and Disease entities. While people often use the two interchangeably in common parlance, the distinction is vital for epidemiological tracking.
  • Length Matters: As shown in the performance graphs, models performed significantly better on shorter tweets (under 40 tokens). As tweet length increases, the complexity of entity relationships grows, leading to a noticeable performance drop.
  • TSA Performance: For targeted sentiment, the best-performing architecture was a combination of depGCN (a graph convolutional network) and COVID-TWITTER-BERT, reaching an overall F1 score of 71.39.

Limitations and Future Work

Despite these strong baselines, F1 scores for Disease and Organization entities remain suboptimal, suggesting that METS-CoV is a challenging dataset that will drive future innovation.

The authors note a few limitations, such as entity imbalance (symptoms appear much more frequently than drugs or vaccines due to the initial filtering lexicon) and the inherent subjectivity of sentiment analysis. Future work should focus on more robust TSA models that can better distinguish between subtle sentiment polarities and techniques to handle the long-tail distribution of rarer medical entities.

Source Code & Data: The authors have made their data, guidelines, and benchmark models publicly available on GitHub to ensure reproducibility.

Summary updated

Abstract

The COVID-19 pandemic continues to bring up various topics discussed or debated on social media. In order to explore the impact of pandemics on people's lives, it is crucial to understand the public's concerns and attitudes towards pandemic-related entities (e.g., drugs, vaccines) on social media. However, models trained on existing named entity recognition (NER) or targeted sentiment analysis (TSA) datasets have limited ability to understand COVID-19-related social media texts because these datasets are not designed or annotated from a medical perspective. In this paper, we release METS-CoV, a dataset containing medical entities and targeted sentiments from COVID-19 related tweets. METS-CoV contains 10,000 tweets with 7 types of entities, including 4 medical entity types (Disease, Drug, Symptom, and Vaccine) and 3 general entity types (Person, Location, and Organization). To further investigate tweet users' attitudes toward specific entities, 4 types of entities (Person, Organization, Drug, and Vaccine) are selected and annotated with user sentiments, resulting in a targeted sentiment dataset with 9,101 entities (in 5,278 tweets). To the best of our knowledge, METS-CoV is the first dataset to collect medical entities and corresponding sentiments of COVID-19 related tweets. We benchmark the performance of classical machine learning models and state-of-the-art deep learning models on NER and TSA tasks with extensive experiments. Results show that this dataset has vast room for improvement for both NER and TSA tasks. With rich annotations and comprehensive benchmark results, we believe METS-CoV is a fundamental resource for building better medical social media understanding tools and facilitating computational social science research, especially on epidemiological topics. Our data, annotation guidelines, benchmark models, and source code are publicly available (\url{https://github.com/YLab-Open/METS-CoV}) to ensure reproducibility.
BibTeX
@inproceedings{zhou2022mets,
  author = {Peilin Zhou and Zeqiang Wang and Dading Chong and Zhijiang Guo and Yining Hua and Zichang Su and Zhiyang Teng and Jiageng Wu and Jie Yang},
  title = {{METS-CoV: A Dataset of Medical Entity and Targeted Sentiment on COVID-19 Related Tweets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=Qo-Enz4xMb},
}
Figure