UKYNLP@SMM4H2024: Language Model Methods for Health Entity Tagging and Classification on Social Media (Tasks 4 & 5)

Motasem Obeidat, Vinu Ekanayake, Md Sultan Al Nahian, Ramakanth KavuluruACL 2024

TL;DR

This paper presents a novel approach using advanced language models for two critical health-related social media analysis tasks: extracting clinical/social impacts of non-medical substance use from Reddit and classifying tweets about children's medical disorders. The authors achieved top performance on the substance use task and a high score on the children's disorder task, demonstrating the power of fine-tuned encoder-only language models for nuanced health information extraction.

What Problem Does This Paper Address?

The paper tackles two significant challenges in leveraging social media for public health research:

  1. Understanding the Nuances of Non-Medical Substance Use: While social media offers a rich source of user-generated content, extracting specific clinical and social impacts of non-medical substance use from platforms like Reddit is complex. This information is crucial for developing targeted interventions and educational programs to mitigate the adverse effects of substance misuse.
  2. Accurate Identification of Children's Health Disorders: Distinguishing between tweets where users report a child having a specific disorder (like ADHD, ASD, delayed speech, or asthma) versus those that merely mention these conditions without personal impact is vital. This distinction is important for data collection and understanding the prevalence and personal impact of these disorders, complementing traditional epidemiological methods.

Methods

The authors employed state-of-the-art language models for both tasks:

  • Task 4 (Substance Use Impact Extraction): This task was framed as a Named Entity Recognition (NER) problem. They utilized a span-based encoder-only model architecture, building upon the Princeton University Relation Extraction (PURE) pipeline. Key to their approach was experimenting with pre-trained models like BERT and ALBERT, and crucially, investigating the impact of adding a bidirectional LSTM (BiLSTM) layer between the span embeddings and the classification layers. This BiLSTM layer was found to improve performance in prior work and was explored here to enhance feature representation.
  • Task 5 (Children's Disorder Classification): For this binary classification task, the team fine-tuned several pre-trained encoder-only language models. Specifically, they used DeBERTa v2 xlarge, DeBERTa v3 Large (pre-trained on MNLI), and RoBERTa base (pre-trained on a general tweet dataset). To further boost performance, they integrated these models into a 3-model majority vote ensemble.

Datasets

The paper utilizes datasets from the 9th Social Media Mining for Health Research and Applications (SMM4H) 2024 workshop:

  • Task 4 Dataset: Consisted of 26,126 Reddit posts. A significant challenge noted was the high class imbalance, with only about 1.22% of posts annotated for clinical or social impacts.
  • Task 5 Dataset: Comprised 10,734 tweets, divided into training (7,398), validation (389), and testing (1,947) sets. This dataset specifically targeted discussions where users reported pregnancy and mentioned health conditions affecting their children.

Key Insights and Ideas

  • BiLSTM Boosts NER Performance (with caveats): For Task 4, the addition of a BiLSTM layer generally improved performance, particularly when combined with the BERT model and a zero context window, achieving the highest F1 score. However, the effectiveness was noted to be dependent on the specific model architecture and task requirements.
  • Context Window Matters for BERT: For Task 4, increasing the context window for the BERT model without a BiLSTM layer actually decreased performance, suggesting that simply providing more text isn't always beneficial without the right architectural support.
  • Ensembling for Classification: For Task 5, combining multiple fine-tuned models (DeBERTa v3, DeBERTa v2, and RoBERTa) through a majority vote ensemble proved effective, achieving strong results.
  • Validation vs. Test Score Discrepancies: The authors observed nontrivial variations in model rankings between validation and test scores for Task 4, highlighting that validation results might not always be strong indicators of final performance, especially with smaller datasets.
  • Informal Language is a Hurdle: Preliminary error analysis for both tasks pointed to informal or casual language, which deviates from grammatical norms, as a significant contributor to both false positives and false negatives.

Key Results

  • Task 4: The submission achieved the top score on Task 4 for clinical and social impact recognition. The best performing BERT model with a BiLSTM layer and zero context window achieved a strict F1 score of 21.5 on the validation set. On the test set, the BERT model without a BiLSTM layer and a context window of 100 achieved a relaxed F1 score of 46.2.
  • Task 5: The submission achieved a high score on Task 5. The DeBERTa v3 Large model (fine-tuned on MNLI) achieved an F1 score of 95.6 on the validation set. On the test set, the DeBERTa v2 XL model achieved the highest F1 score of 94.8, with the ensemble model close behind at 94.2.

Limitations and Future Work

  • Informal Language Mitigation: The authors plan to address the challenge of informal language by exploring strategies like lowering output probability thresholds and investigating more sophisticated loss functions and post-processing techniques.
  • LLM Exploration: They acknowledge not using decoder-only Large Language Models (LLMs) like Mistral or Llama due to prior lab findings favoring encoder-only models for information extraction. However, they propose reassessing this with larger LLMs (7B+ parameters) to see if they offer benefits, especially for shorter entity NER tasks.
  • Data Augmentation: A future direction includes exploring the use of larger LLMs (e.g., GPT-4) for generating synthetic examples to augment training data for both tasks, though challenges remain in preserving long named entities intact during augmentation for NER.

Summary updated

BibTeX
@inproceedings{obeidat2024ukynlp,
  author = {Motasem Obeidat and Vinu Ekanayake and Md Sultan Al Nahian and Ramakanth Kavuluru},
  title = {{UKYNLP@SMM4H2024: Language Model Methods for Health Entity Tagging and Classification on Social Media (Tasks 4 \& 5)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure