NCUEE-NLP at SemEval-2023 Task 8: Identifying Medical Causal Claims and Extracting PIO Frames Using the Transformer Models

Lung-Hao Lee, Yuan-Hao Cheng, Jen-Hao Yang, Kao-Yuan TienACL 2023

TL;DR

This paper presents the NCUEE-NLP system for SemEval-2023 Task 8, which aims to automatically identify medical causal claims and extract key elements (Population, Intervention, Outcome) from Reddit posts. Their DeBERTa-based approach achieved strong results, ranking second overall in both subtasks, demonstrating the effectiveness of advanced transformer models for this challenging medical NLP task.

What Problem Does This Paper Address?

The paper tackles the crucial task of automatically understanding and extracting structured information from medical text, specifically focusing on causal claims found in online discussions like Reddit. Identifying these claims and their associated components (who is affected, what is being done, and what is the result) is vital for various applications, including clinical decision support, public health monitoring, and evidence-based medicine. While previous work has focused on structured medical literature (like PubMed), this task extends the challenge to the more informal and noisy domain of social media, where identifying nuanced causal relationships and PICO (Population, Intervention, Comparison, Outcome) frames is significantly more difficult. The SemEval-2023 Task 8 specifically addresses this by asking participants to identify medical causal claims and extract PIO frames from Reddit posts.

Methods

The NCUEE-NLP system leverages pre-trained transformer models, with a particular focus on DeBERTa (Decoding-enhanced BERT with disentangled attention) and its improved version, DeBERTaV3.

For Subtask 1: Causal Claim Identification, the system treats it as a sentence-level classification task. Each Reddit post is first segmented into sentences using the Trankit toolkit. Then, a fine-tuned DeBERTa model classifies each sentence into one of five categories: "Claim," "Experience," "Experience based claim," "Question," or "Others." The final sentence category is determined by a majority vote of its constituent tokens.

For Subtask 2: PIO Frame Extraction, the system employs a weakly supervised sequence labeling approach using DeBERTa. A special token "@" is used to mark the start and end of a claim. The model then labels each token within a claim as belonging to "Population" (Pop-tag), "Intervention" (Int-tag), "Outcome" (Out-tag), or "null" if it doesn't belong to any of these frames.

Datasets

The study utilizes datasets provided by the SemEval-2023 Task 8 organizers, which are derived from Reddit posts. Due to privacy concerns, the organizers provide post identifiers, annotations, and a script to retrieve the actual post content. The paper notes that some posts may have been deleted by users, leading to missing data.

  • Subtask 1 Training Data: 5,287 posts (88.6% of provided identifiers), resulting in 43,237 sentences after segmentation.
  • Subtask 1 Test Data: 1,264 posts (88.8% of provided identifiers), resulting in 11,492 sentences.
  • Subtask 2 Training Data: 524 posts (87.8% of provided identifiers).
  • Subtask 2 Test Data: 130 posts (86.7% of provided identifiers).

The paper also mentions using publicly available pre-trained models like BERT, BioBERT, and RoBERTa for comparison.

Key Insights and Ideas

  • DeBERTa's Superiority: The DeBERTa transformer, with its disentangled attention and enhanced mask decoder, consistently outperformed other transformer models (BERT, BioBERT, RoBERTa) in the 5-fold cross-validation on the training set for both subtasks. This highlights the architectural advantages of DeBERTa for complex NLP tasks.
  • Domain Adaptation Matters: While BioBERT showed slight improvements over BERT, likely due to its biomedical pre-training, DeBERTa's general advancements in transformer architecture proved more impactful for this specific task.
  • Social Media Noise is a Challenge: The performance on the test set, while strong, indicates the inherent difficulty of extracting structured information from the less formal and potentially noisier domain of Reddit compared to structured medical literature.
  • Intervention Extraction is Particularly Hard: For PIO frame extraction, the Intervention class showed the highest F1-score for DeBERTaV3 (50.72%) in the training set, suggesting that identifying the "what is being done" aspect might be more straightforward than other elements in this context.

Key Results

The NCUEE-NLP system achieved impressive results, ranking second overall among participating teams in both subtasks:

  • Subtask 1 (Causal Claim Identification): The best submission achieved an F1-score of 70.15%.
  • Subtask 2 (PIO Frame Extraction): The system achieved class-wise F1-scores of 37.78% for Population, 43.58% for Intervention, and 30.67% for Outcome, resulting in a macro-averaged F1-score of 37.34%.

Limitations and Future Work

  • Data Scarcity for Subtask 2: The paper acknowledges that the datasets for Subtask 2 (PIO frame extraction) are relatively small, which can limit the robustness of the models.
  • Deleted Posts: The issue of deleted Reddit posts impacting data availability for training and testing is a practical limitation of working with user-generated content.
  • Performance Gap: While the system performed well, there's still room for improvement, particularly in the PIO frame extraction subtask, where the macro-averaged F1-score is below 40%.

Future work could involve exploring larger and more diverse datasets, investigating more advanced weak supervision techniques, and potentially incorporating external medical knowledge bases to further enhance the accuracy of causal claim identification and PIO frame extraction.

Summary updated

BibTeX
@inproceedings{lee2023ncuee,
  author = {Lung-Hao Lee and Yuan-Hao Cheng and Jen-Hao Yang and Kao-Yuan Tien},
  title = {{NCUEE-NLP at SemEval-2023 Task 8: Identifying Medical Causal Claims and Extracting PIO Frames Using the Transformer Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure