HBUT at #SMM4H 2024 Task1: Extraction and Normalization of Adverse Drug Events with a Large Language Model

Yuanzhi Ke, Hanbo Jin, Xinyun Wu, Caiquan XiongACL 2024

TL;DR

This paper presents a novel two-step approach using the GLM large language model for extracting adverse drug events (ADEs) from social media text and normalizing them to MedDRA preferred terms. Their method achieves competitive recall rates, outperforming the average and median participant scores, demonstrating the effectiveness of LLMs with carefully crafted prompts for this challenging task.

What Problem Does This Paper Address?

Extracting and normalizing adverse drug events (ADEs) from social media is a crucial but challenging task. While models like BERT and its variants (BioBERT, ClinicalBERT) have shown promise in named entity recognition (NER), they often require extensive fine-tuning data, which is scarce for specialized domains like social media health data. This paper tackles the problem of identifying ADEs from Twitter (X) posts and mapping them to standardized medical terms (MedDRA preferred terms) efficiently, even with limited training data, by leveraging the few-shot learning capabilities of large language models (LLMs).

Methods

The authors employ a two-phase system: ADE Text Extraction and Normalization.

ADE Text Extraction: 1. Potential ADE Extraction: They use the GLM-4 LLM with a few-shot in-context learning approach. This involves providing the model with examples of tweets and their corresponding ADEs to guide it in identifying potential ADE mentions. 2. Self-Improvement (Filtering): To reduce false positives, a second step uses a question-and-answer style prompt. The LLM is asked to determine if the extracted ADEs are actually caused by a drug mentioned in the tweet, rather than other factors. This acts as a self-consistency check.

Normalization to MedDRA ptIDs: An ensemble method is used for normalization. This involves:

  • Embedding Models: Five pre-trained embedding models (all-MiniLm-L6-v2, PubMedBERT-base-embeddings, BioBERT-NLI, BioBERT-mnli-snli-scinli-scitail-mednli-stsb, and ClinicalBERT) are used to generate embeddings for the extracted ADE text and MedDRA terms.
  • Levenshtein Distance: This metric is used to measure the string similarity between extracted ADEs and MedDRA terms.
  • Reciprocal Rank Fusion (RRF): The relevance scores from the embedding models and Levenshtein Distance are fused using RRF. This method combines the rank-based scores from multiple sources to produce a final score, which is then used to select the most relevant MedDRA preferred term ID (ptID).

Datasets

The primary dataset used is the SMM4H 2024 Task 1 dataset, which consists of Twitter (X) posts. The paper mentions using this dataset for evaluation. For normalization, the MedDRA terminology is used as the target vocabulary. The authors also mention using their local experiments on a DEV set for model selection and comparison.

Key Insights and Ideas

  • Few-Shot Learning with LLMs is Powerful: The paper strongly advocates for the effectiveness of few-shot learning with LLMs like GLM for tasks like ADE extraction, especially when labeled data is scarce. The ability to achieve good performance with minimal examples is a significant advantage.
  • Prompt Engineering is Crucial: The success of their system hinges on carefully designed prompts for both the extraction and self-improvement steps. The authors highlight that the language of the prompt (even in Chinese for GLM) significantly impacts performance.
  • Two-Step Extraction Improves Precision: The self-improvement step, which filters out non-drug-related events, is key to reducing false positives and improving the overall quality of extracted ADEs.
  • Ensemble Normalization Enhances Robustness: Combining multiple embedding models and Levenshtein distance via RRF provides a more robust normalization mechanism, likely capturing different facets of similarity.
  • High Recall is Achievable: The system demonstrates strong performance in recall, indicating its ability to identify a large proportion of actual ADEs, which is often a primary goal in such tasks.

Key Results

The system achieved the following recall rates on the test dataset:

  • ADE Normalization Recall (R-Norm): 41.1%
  • ADE Recognition Recall (R-NER): 42.8%
  • Normalization for Unseen ADEs Recall (R-Norm-Unseen): 40.6%

These scores were generally 10%-20% higher than the median and average scores of other participants. The authors note that their method achieved excellent recall without requiring extensive fine-tuning on downstream tasks.

Limitations and Future Work

Limitations: * Precision Decline: The paper acknowledges a decline in the precision of ADE mining. They attribute this to the potential mismatch between the general knowledge of LLMs and the specific, often informal, language used in social media ADE reports. This gap can lead to misidentification.

Future Work: * Improving Precision: Addressing the precision issue is identified as a key area for future work. This might involve more sophisticated prompt design, domain-specific fine-tuning (if data becomes available), or exploring other methods to better bridge the gap between general LLM knowledge and social media discourse.

Summary updated

BibTeX
@inproceedings{ke2024hbut,
  author = {Yuanzhi Ke and Hanbo Jin and Xinyun Wu and Caiquan Xiong},
  title = {{HBUT at \#SMM4H 2024 Task1: Extraction and Normalization of Adverse Drug Events with a Large Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure