Enhancing Social Media Health Prediction Certainty by Integrating Large Language Models with Transformer Classifiers

Sedigh Khademi, Christopher Palmer, Muhammad Javed, Jim Buttery, Gerardo DimaguilaACL 2024

TL;DR

This paper tackles the challenge of accurately identifying specific child health conditions (ADHD, ASD, delayed speech, asthma) in noisy social media text. They propose a novel pipeline that combines the strengths of transformer classifiers with the reasoning power of large language models (LLMs) like GPT-4 to improve prediction certainty, especially for ambiguous cases.

What Problem Does This Paper Address?

Social media is a rich source of information about public health, but extracting reliable insights is difficult. Lay language, misspellings, abbreviations, and the sheer volume of data make it hard for traditional NLP models to accurately identify specific health conditions. This paper focuses on a specific SMM4H task: identifying tweets where users discuss their child's ADHD, ASD, delayed speech, or asthma, distinguishing these personal accounts from general discussions. The core challenge is dealing with the inherent noise and ambiguity in social media text to achieve high prediction certainty.

Methods

The authors developed a three-step pipeline:

  1. Data Preparation & Imbalance Handling: They addressed the imbalanced training dataset (more negative than positive examples) using a topic-modeling-based under-sampling technique. This involved identifying repetitive topics and strategically sampling negative examples based on their proximity to topic centroids, ensuring diversity. They also relabeled some ambiguous negative records as positive.
  2. Transformer Classifier Training: They trained RoBERTa-based classifiers on the balanced dataset. This is a standard but powerful approach for text classification.
  3. LLM Refinement: The key innovation lies here. For tweets where the transformer classifier's confidence was below a threshold (0.9), they employed GPT-4 to refine the prediction. This leverages the LLM's advanced reasoning capabilities to handle uncertain cases. They specifically chose GPT-4's zero-shot Chain-of-Thought (CoT) prompting strategy for its superior performance.

Datasets

  • Training Data: Twitter data for SMM4H 2024 Task 5, consisting of 7,398 tweets (2,280 positive, 5,118 negative).
  • Validation Data: 389 tweets (135 positive, 254 negative).
  • Test Data: 10,000 unlabelled records provided after model building and validation.

The paper does not introduce a new dataset but utilizes the competition's provided benchmark.

Key Insights and Ideas

  • LLMs as "Arbiters" for Uncertainty: The most significant insight is that LLMs, particularly GPT-4, can act as powerful "arbiters" to refine predictions from transformer models in cases of low confidence. This hybrid approach leverages the strengths of both model types.
  • The Power of Zero-Shot CoT Prompting: The effectiveness of GPT-4's zero-shot Chain-of-Thought prompting for complex reasoning tasks on social media text is highlighted. This suggests that carefully crafted prompts can unlock LLMs' potential without extensive task-specific fine-tuning.
  • Data Imbalance Mitigation is Crucial: The paper demonstrates that addressing data imbalance through techniques like topic-modeling-based under-sampling is essential for building robust classifiers.

Key Results

  • The proposed LLM+Clf pipeline achieved an F1 score of 0.956 on the validation set, a 2 percentage point improvement over the standalone RoBERTa classifier (0.938).
  • On the test dataset, the LLM+Clf approach yielded an F1 score of 0.927, outperforming the median competitor results by almost 3 percentage points.
  • The best standalone BERTweet model achieved an F1 score of 0.912, and the best Twitter-RoBERTa model reached 0.938.

Limitations and Future Work

  • LLM Limitations: The authors acknowledge that LLMs on their own may not be ideal for all classification tasks due to limitations in overall accuracy, efficiency, and resource requirements.
  • Future Work: While not explicitly detailed, the success of the hybrid approach suggests future work could explore other LLMs, different prompting strategies, and more sophisticated methods for integrating LLM outputs with transformer predictions. Further research into optimizing LLM usage for efficiency and cost-effectiveness would also be valuable.

Summary updated

BibTeX
@inproceedings{khademi2024enhancing,
  author = {Sedigh Khademi and Christopher Palmer and Muhammad Javed and Jim Buttery and Gerardo Dimaguila},
  title = {{Enhancing Social Media Health Prediction Certainty by Integrating Large Language Models with Transformer Classifiers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure