RIGA at SMM4H-2024 Task 1: Enhancing ADE discovery with GPT-4

Eduards Mukans, Guntis BarzdinsACL 2024

TL;DR

This paper presents the RIGA team's approach to extracting and normalizing adverse drug events (ADEs) from social media. They leverage GPT-4 for data augmentation and to enhance existing pipeline components, achieving performance comparable to the state-of-the-art baseline and significantly outperforming mean and median submissions.

What Problem Does This Paper Address?

The SMM4H-2024 Task 1 challenges participants to extract and normalize adverse drug events (ADEs) from English tweets, mapping them to MedDRA high-level term identifiers (HLTIs). This task is crucial for pharmacovigilance, enabling the monitoring of drug safety through real-world data from social media. However, social media data presents challenges such as high imbalance (most tweets don't contain ADEs), the need for precise span extraction, and the complexity of mapping extracted mentions to standardized medical terms. Existing methods often rely on multi-component pipelines, and the introduction of unseen HLTIs in the test set further complicates the normalization step.

Methods

The RIGA team employs a three-stage pipeline similar to the task baseline:

  1. Tweet Classification: To address the data imbalance, they first use a binary classification model (fine-tuned RoBERTa-large) to identify tweets containing ADEs. Crucially, before fine-tuning, they use GPT-4 Turbo to extract potential ADEs from the tweets. This GPT-generated output is then concatenated with the original tweet as input for the classifier.
  2. ADE Span Extraction: For tweets classified as containing ADEs, a Named Entity Recognition (NER) model is used to extract the precise spans of text referring to ADEs. They utilize a BIO-tagging schema. Similar to the classification stage, GPT-4's output is incorporated as additional context for fine-tuning the DeBERTaV3 model. This "Custom + GPT" approach aims to reduce noise and improve accuracy compared to using GPT's raw output.
  3. Span Mapping to MedDRA HLTIs: To map extracted ADE spans to MedDRA HLTIs, they leverage OpenAI's Embedding API. They compute embeddings for all MedDRA HLTIs and for each extracted ADE span, then identify the closest HLTIs using cosine similarity. This approach is chosen due to the large number of HLTIs and the presence of unseen categories in the test set, making traditional classification infeasible.

Datasets

The paper uses the SMM4H-2024 Task 1 dataset, which includes train, development, and test splits. The dataset is characterized by a high imbalance, with only 6.8% of training tweets and 6.7% of development tweets containing ADEs. The "Full" dataset contains 18,185 training tweets and 965 development tweets, while the "Contain ADES" subset has 1,239 training and 65 development tweets. The test set size is 11,799 tweets. A key difference from previous versions of the task is the inclusion of negative samples (mentions not falling under MedDRA categories) in the dataset splits.

Key Insights and Ideas

  • GPT-4 as a Data Augmenter: The core idea is to use GPT-4 not just for direct extraction but as a powerful tool to augment existing data for fine-tuning downstream models. This is particularly effective for tasks with limited labeled data or complex entity recognition.
  • Pipeline Enhancement: Integrating GPT-generated context into each stage of a traditional pipeline (classification, NER) significantly boosts performance, demonstrating the synergistic effect of LLMs with established NLP architectures.
  • Addressing Data Imbalance: The multi-stage approach, starting with a robust tweet classifier, is essential for efficiently handling the vast majority of non-ADE tweets and focusing computational resources on relevant data.
  • Embedding-based Normalization: Using embeddings for span-to-HLTI mapping proves to be a viable strategy for handling a large and potentially unseen set of target entities, circumventing the limitations of traditional classification.

Key Results

The RIGA team's "Custom + GPT" submission achieved performance comparable to the task's baseline on the F1-NER metric and significantly outperformed the mean and median submissions. The "GPT few-shot" submission also showed results above the mean and median for both F1-Norm and F1-NER metrics. While the "Custom + GPT" model's F1-Norm score suffered, its overall performance was strong, especially in ADE span extraction. The embedding-based normalization approach was effective, though the authors note they ran out of resources to further optimize it.

Limitations and Future Work

The authors acknowledge several limitations:

  • GPT Hallucinations and Verbosity: GPT-generated spans can sometimes include irrelevant verbs or exhibit verbosity.
  • Textual Deviations: GPT might generate ADE expressions that don't precisely match the original tweet text, potentially due to biases in its training data (e.g., American dialects).
  • Resource Constraints: The team ran out of resources and time to further optimize the "Custom + GPT" model and achieve higher F1-Norm scores.
  • "Summary" Field Ineffectiveness: The initial idea of using a "Summary" field in the classification prompt to simplify the task proved ineffective, suggesting that a "rewrite" instruction might be more beneficial.

Future work could involve exploring "rewrite" prompts for classification, further refining the GPT integration into the pipeline, and potentially investigating more advanced embedding techniques or hybrid approaches for the normalization stage.

Summary updated

BibTeX
@inproceedings{mukans2024riga,
  author = {Eduards Mukans and Guntis Barzdins},
  title = {{RIGA at SMM4H-2024 Task 1: Enhancing ADE discovery with GPT-4}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure