TLab at #SMM4H 2024: Retrieval-Augmented Generation for ADE Extraction and Normalization

Jacob Berkowitz, Apoorva Srinivasan, Jose Miguel Acitores Cortina, Nicholas P TatonettiACL 2024

TL;DR

This paper introduces a novel Retrieval-Augmented Generation (RAG) approach for identifying and standardizing adverse drug events (ADEs) from social media. By combining powerful language models with targeted information retrieval, the method effectively handles the informal language of tweets, even for drug terms not seen during training, paving the way for more robust public health monitoring.

What Problem Does This Paper Address?

Social media platforms like X (formerly Twitter) are a goldmine of real-time information about how people experience medications. However, extracting reliable data on Adverse Drug Events (ADEs) from these platforms is a significant challenge. The language used is often informal, diverse, and can be difficult for traditional methods, which typically require massive annotated datasets and extensive training. This reliance on large datasets is computationally expensive and may not capture the ever-evolving nature of social media language. The SMM4H 2024 Task 1 specifically aims to tackle this by identifying ADEs in tweets and mapping them to standardized MedDRA Preferred Terms (PTs).

Methods

The TLab team developed a two-pronged RAG approach:

  1. ADE Tweet Classification: For initial identification of tweets containing ADEs, they employed a fine-tuned BERT-large-uncased model. This is a standard transformer-based approach, but the novelty lies in its application to this specific, challenging domain.
  2. Retrieval-Augmented Generation (RAG) for Extraction and Normalization: This is the core innovation.
  3. Extraction: For tweets identified as containing ADEs, they used the SFR-embedding-mistral model to create embeddings. These embeddings are stored in a vector database. When a new tweet is processed, its embedding is compared to the database to retrieve the top 10 most semantically similar tweets. This "few-shot prompting" technique provides contextual examples to a large language model. They then used Meta's Llama 3 7b model, prompted with these retrieved examples and the original tweet, to extract ADE terms and their text spans. This choice of Llama 3 was motivated by cost considerations compared to GPT-4 for this step.
  4. Normalization: To map the extracted ADE terms to standardized MedDRA PTs, they again used the SFR-embedding-mistral model to embed the extracted terms and compared them to a database of MedDRA PT embeddings. The top 10 most similar MedDRA PTs were retrieved. Finally, GPT-4 was used to process the original tweet along with these potential MedDRA PT matches, leveraging the tweet's context to select the most appropriate standardized term.

Datasets

The paper primarily discusses the use of the SMM4H 2024 Task 1 dataset, which includes annotated tweets for ADE detection and normalization. The performance is evaluated against task-specific metrics and compared to the mean and median scores of other participants. No new datasets were introduced.

Key Insights and Ideas

  • RAG as a Data-Efficient Solution: The paper strongly advocates for RAG as a way to bypass the need for massive, task-specific annotated datasets. By retrieving relevant examples on-the-fly, the models can learn from a few high-quality examples, making the process more efficient and adaptable.
  • Leveraging Diverse LLMs: The approach effectively combines the strengths of different models: BERT for initial classification, SFR-embedding-mistral for semantic understanding and retrieval, Llama 3 for extraction, and GPT-4 for nuanced normalization. This "ensemble" of models, orchestrated by RAG, is key to their success.
  • Robustness to Unseen Terms: A significant finding is the model's strong performance on normalizing MedDRA PTs that were not present in the training data. This highlights the power of RAG in generalizing beyond explicitly trained examples, which is crucial for real-world applications where new drug-related terms emerge constantly.

Key Results

The proposed RAG method achieved competitive performance on the SMM4H 2024 Task 1:

  • Named Entity Recognition (NER): An F1 score of 0.392, surpassing the task mean (0.327) and median (0.376).
  • Text Normalization: An F1 score of 0.359, outperforming the task mean (0.283) and median (0.293).
  • Normalization on Unseen Data: A notable F1 score of 0.363, significantly exceeding the task mean and median (both 0.209). This demonstrates exceptional robustness in handling novel terms.

Limitations and Future Work

The authors acknowledge several areas for future exploration:

  • Arbitrary Retrieval Number: The choice of retrieving the top 10 similar examples was arbitrary. Further validation could explore optimal numbers for ADE extraction and normalization.
  • Exploring More Few-Shot Techniques: Investigating additional few-shot learning techniques and retrieval methods could further enhance adaptability to evolving social media language.
  • GPT-4 for NER: While cost-prohibitive for extensive use, evaluating GPT-4 for the NER task could potentially yield performance improvements.
  • Fine-tuning Local LLMs: For scenarios with abundant training data, fine-tuning a local LLM (like Llama 3) for both NER and normalization could be a viable alternative.

Summary updated

BibTeX
@inproceedings{berkowitz2024tlab,
  author = {Jacob Berkowitz and Apoorva Srinivasan and Jose Miguel Acitores Cortina and Nicholas P Tatonetti},
  title = {{TLab at \#SMM4H 2024: Retrieval-Augmented Generation for ADE Extraction and Normalization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure