MasonNLP+ at SemEval-2023 Task 8: Extracting Medical Questions, Experiences and Claims from Social Media using Knowledge-Augmented Pre-trained Language Models

Giridhar Kaushik Ramachandran, Haritha Gangavarapu, Kevin Lybarger, Ozlem UzunerACL 2023

TL;DR

This paper presents a novel approach for extracting medical questions, patient experiences, and condition-related entities from social media platforms like Reddit. By leveraging knowledge-augmented pre-trained language models, the authors achieved a 3rd place ranking in the SemEval-2023 Task 8 challenge, demonstrating the effectiveness of their method for understanding complex medical discussions online.

What Problem Does This Paper Address?

Social media platforms have become a rich source of information about medical conditions, treatments, and patient experiences. However, this information is often unstructured and can be inconsistent with scientific evidence, making it challenging to monitor for misinformation or gain insights into real-world health outcomes. The SemEval-2023 Task 8 aimed to address this by focusing on extracting specific types of medical information from Reddit posts: patient experiences (questions, claims) and medical condition details (population, intervention, outcome). The challenge lies in accurately identifying and categorizing these diverse entities within the noisy and informal language of social media.

Methods

The core of the MasonNLP+ approach relies on Bidirectional Encoder Representations from Transformer (BERT)-based systems. They experimented with several strategies:

  • Fine-tuning General and Mixed-Domain Models: They fine-tuned pre-trained language models like BERT and RoBERTa, as well as domain-specific models like BioMedRoBERTa and BioRedditBERT, on the medical text data. This leverages existing knowledge from large text corpora and adapts it to the medical domain.
  • Knowledge Augmentation: To improve entity extraction, they incorporated external knowledge about diseases and chemicals. This was achieved through a data augmentation technique where identified disease and chemical entities were encoded with special tokens. This augmentation was performed using a pre-trained model (en_ner_bc5cdr_md) trained on a biomedical corpus (BC5CDR).
  • Sequence Tagging: Both subtasks were framed as sequence tagging problems, where each token in a post was assigned a label (e.g., Begin-Inside-Outside - BIO tagging).

Datasets

The primary dataset used in this work is the RedHot corpus, specifically a subset curated for the SemEval-2023 Task 8. This corpus contains over 22,000 Reddit posts from 24 medical condition-related subreddits.

  • Subtask-1: Focused on extracting patient experiences, including claims, personal experiences, and questions. The training set for this subtask contained 4,964 posts.
  • Subtask-2: Focused on extracting medical condition information, specifically population, intervention, and outcome entities. The training set for this subtask contained 517 posts.

The authors also mention using the BC5CDR corpus for training the knowledge augmentation model.

Key Insights and Ideas

  • Domain-Specific Models Show Promise: While general-domain models performed well on patient experience extraction, domain-specific models like BioRedditBERT showed a significant advantage, particularly in detecting population entities. This highlights the importance of pre-training on relevant data.
  • Knowledge Augmentation Benefits are Nuanced: Augmenting models with external disease and chemical knowledge did not always lead to significant improvements, especially for longer entity spans like personal experiences. However, it did show benefits for shorter, more structured entities like population and intervention, which often directly overlap with disease or chemical names.
  • Inter-Annotator Agreement Matters: The paper notes that lower inter-annotator agreement for certain entity types (like claims and experiences) can make it harder for models to achieve high performance, as there's inherent ambiguity in the gold labels themselves.
  • Error Analysis Reveals Key Challenges: The error analysis highlighted issues with sentence tokenization (splitting long descriptions), confusability between similar entity types (claim vs. personal experience), and challenges in extracting abbreviated medical conditions.

Key Results

  • The MasonNLP+ system achieved 3rd place on the leaderboards for both Subtask-1 and Subtask-2 of the SemEval-2023 Task 8.
  • For Subtask-1 (patient experience), their system achieved an F1 score of 68.59% at the sentence level and 32.65% at the token level on the challenge test set.
  • For Subtask-2 (medical conditions), their system achieved an F1 score of 20.83% for intervention entities and 23.27% for outcome entities on the test set.
  • On the validation set, the BioRedditBERT+ model (domain-specific with knowledge augmentation) showed the best performance for Subtask-1, achieving an F1 score of 49.52%. For Subtask-2, BioRedditBERT+ also performed well, with an F1 score of 27.31% for outcome entities.

Limitations and Future Work

  • Limitations:
    • The performance on Subtask-2 (PIO entities) was generally lower than Subtask-1, indicating a greater challenge in extracting these more granular medical condition details.
  • The effectiveness of knowledge augmentation was limited for longer, more ambiguous entity spans.
  • Low inter-annotator agreement for certain entity types poses a fundamental challenge.
  • Future Work:
    • The authors suggest utilizing cross-sentence information to better handle long descriptions and context.
  • Entity normalization is proposed as a way to improve the extraction of experience-related and medical condition-related entities.
  • Further exploration of knowledge-augmented systems and their application in practical scenarios is recommended.

Summary updated

BibTeX
@inproceedings{ramachandran2023masonnlp,
  author = {Giridhar Kaushik Ramachandran and Haritha Gangavarapu and Kevin Lybarger and Ozlem Uzuner},
  title = {{MasonNLP+ at SemEval-2023 Task 8: Extracting Medical Questions, Experiences and Claims from Social Media using Knowledge-Augmented Pre-trained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure