Claim Detection in Biomedical Twitter Posts

Amelie Wührl, Roman KlingerNAACL 2021

TL;DR

This paper introduces the first annotated dataset for detecting biomedical claims on Twitter, distinguishing between explicit and implicit claims. They demonstrate that while claim detection is challenging in this domain, baseline models achieve acceptable performance, particularly for explicit claims, and highlight the difficulty of generalizing models across different domains.

What Problem Does This Paper Address?

Social media platforms like Twitter are rich sources of information, including valuable insights into medical conditions, treatments, and patient experiences. However, this also means they can be a breeding ground for misinformation, which can be particularly harmful in the biomedical sphere. While automatic fact-checking and fake news detection methods exist, they haven't been extensively applied to the unique challenges of biomedical discourse on social media. This paper aims to bridge this gap by developing methods to automatically identify "claims" – statements presented as factual conclusions – within biomedical tweets. This is a crucial first step for downstream tasks like fact-checking and hypothesis generation.

Methods

The authors tackle claim detection using a classification framework. They explore two main settings:

  • Binary Classification: Distinguishing between tweets that contain a claim and those that don't.
  • Multi-class Classification: Differentiating between explicit claims, implicit claims, and non-claims. They also propose a Multi-class Pipeline approach, where a first classifier identifies claims, and a second binary classifier distinguishes between explicit and implicit claims.

For their models, they employ standard text classification techniques:

  • Naive Bayes (NB)
  • Logistic Regression (LG)
  • Bidirectional Long Short-Term Memory (BiLSTM) networks
  • BERT (Bidirectional Encoder Representations from Transformers), a powerful pre-trained language model, which they fine-tune on their specific dataset.

These models are trained using 50-dimensional FastText embeddings (for NB, LG, BiLSTM) or directly fine-tuned (for BERT).

Datasets

The core contribution of this paper is the creation of a novel, manually annotated Twitter corpus.

  • Biomedical Claim Twitter Corpus: This corpus consists of 1200 tweets related to four medical topics: COVID-19, measles, cystic fibrosis, and depression. The tweets were collected using a keyword-based retrieval system from the Twitter API, employing four categories of search terms (disease names/hashtags, topical hashtags, disease + treatment/cure terms, and drug names). The corpus is annotated with:
  • A binary label indicating whether a tweet contains a claim.
  • A label for explicitness (explicit or implicit) if a claim is present.
  • Span annotations for the exact claim phrase in explicit claims.

This dataset is made publicly available.

Key Insights and Ideas

  • Biomedical Tweets are Claim-Dense: A significant portion (44.75%) of the collected biomedical tweets contain claims, indicating the prevalence of argumentative or assertive statements in this domain.
  • Implicit Claims are Tough: Detecting implicit claims is considerably more challenging than explicit ones, even for advanced models like BERT. This suggests that inferring claims requires a deeper understanding of context, nuance, and potentially authorial intent.
  • Domain Specificity Matters: Models trained on general domains (like persuasive essays) do not perform as well when applied to biomedical tweets, and vice-versa. This highlights the unique linguistic characteristics and challenges of biomedical discourse on social media.
  • Formality as a Factor: The difference in formality between text types (e.g., essays vs. tweets) appears to be a significant factor influencing claim detection performance.

Key Results

  • Binary Claim Detection: The LG classifier achieved the best F1 score of 0.70 for detecting claims in a binary setting. BERT and NB performed slightly lower, while BiLSTM showed substantially lower performance.
  • Multi-class Claim Detection: Overall scores were lower in the multi-class setting. The pipeline approach, separating claim/non-claim and then explicit/implicit, showed promise, with the best F1 score for explicit claims reaching 0.59.
  • Implicit Claim Performance: Implicit claim detection consistently lagged behind explicit claim detection across all models and settings, with the best F1 score for implicit claims being 0.36.
  • Cross-domain Performance: When trained on persuasive essays, the LG model achieved an F1 score of 0.98 for claim detection. However, applying this model to tweets resulted in a significant drop in performance (F1 of 0.66), and vice-versa. Combining training data from both domains did not improve performance.

Limitations and Future Work

  • Inter-Annotator Agreement: While the agreement improved throughout guideline development, the span annotation task showed limited agreement (κ=.38), indicating its inherent difficulty.
  • Dataset Size: The authors suggest that the relatively small dataset size might be a reason why more complex models like LSTM and BERT did not consistently outperform simpler models, especially for implicit claims.
  • Argumentative Intention: The LG model struggled with cases where argumentative intention was ambiguous, sometimes misclassifying motivational statements as non-claims.
  • Future Work:
    • Investigating the specific reasons for the challenging nature of biomedical Twitter data.
  • Developing models that generalize better across domains.
  • Exploring the inclusion of annotators with medical expertise to improve annotation quality.
  • Conducting detailed introspection of topics to identify specific claim categories that are harder to label.
  • Integrating claim detection with fact-checking approaches for misinformation detection.

Summary updated

BibTeX
@inproceedings{wuhrl2021claim,
  author = {Amelie Wührl and Roman Klinger},
  title = {{Claim Detection in Biomedical Twitter Posts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure