PolyuCBS at SMM4H 2024: LLM-based Medical Disorder and Adverse Drug Event Detection with Low-rank Adaptation

Zhai Yu, Xiaoyi Bao, Emmanuele Chersoni, Beatrice Portelli, Sophia Lee, Jinghang Gu, Chu-Ren HuangACL 2024

TL;DR

This paper presents LLM-based approaches for detecting medical disorders and adverse drug events in social media text, achieving strong results on the SMM4H 2024 shared tasks. By combining pre-trained language models (PLMs) with large language models (LLMs) and employing parameter-efficient fine-tuning techniques like Low-Rank Adaptation (LoRA), the team effectively tackled the challenges of colloquial language and complex medical information.

What Problem Does This Paper Address?

Social media has become a rich source of health-related information, but extracting accurate medical insights from this noisy data is challenging. Specifically, this paper addresses two tasks from the 9th Social Media Mining for Health Applications (SMM4H) 2024 Shared Tasks:

  1. Adverse Drug Event (ADE) Detection and Normalization: Identifying mentions of ADEs in tweets and mapping them to standardized medical concepts (MedDRA Preferred Term IDs).
  2. Children's Medical Disorder Classification: Distinguishing tweets that report specific medical disorders in children (ADHD, ASD, delayed speech, asthma) from those that merely mention these conditions or where the patient is not a child.

The core challenges include handling colloquial language, abbreviations, and implicit information common in social media, as well as effectively leveraging the knowledge embedded within large language models for these specialized tasks.

Methods

The team employed a combination of PLMs and LLMs, focusing on parameter-efficient fine-tuning to adapt these models to the specific tasks.

  • Task 1 (ADE Detection & Normalization):
    • Extraction: BERT was used as a baseline, with experiments involving domain continual pre-training and fine-tuning using BIO tags. A novel approach involved treating extraction as a "cloze test" task, where the model predicts masked tokens within a sentence template. LLaMA-2 was also explored for this task.
  • Normalization: This was framed as a classification problem, with SapBERT and CODER being the chosen models.
  • Task 5 (Children's Medical Disorder Classification):
    • LLM-based Approach: LLaMA-2-7B was fine-tuned using Low-Rank Adaptation (LoRA). LoRA is a technique that significantly reduces the number of trainable parameters by introducing low-rank decomposition matrices to specific weights, making fine-tuning large models more efficient.
  • Instruction Tuning & Prompt Engineering: To handle the complex requirements of Task 5 (specific disorders, parent reporting, child as patient), a decomposed prompt approach was used. The task was broken down into independent units, each represented by a requirement, and the LLM was fine-tuned to address them sequentially. This involved crafting specific prompts and target sequences to guide the LLM's understanding.

Datasets

The paper utilizes datasets from the SMM4H 2024 Shared Tasks for both Task 1 and Task 5. Specific details about the size and composition of these datasets are not elaborated upon beyond their use in the respective tasks.

Key Insights and Ideas

  • LLMs Outperform PLMs (with caveats): While LLMs like LLaMA-2 showed promise and generally outperformed traditional PLMs like BERT on these tasks, they also exhibited a higher tendency for "hallucination" (generating incorrect or nonsensical outputs).
  • Decomposed Prompts for Complex Tasks: For intricate tasks like Task 5, breaking down the problem into smaller, manageable prompts significantly improved the LLM's ability to understand and address all requirements.
  • LoRA for Efficient LLM Fine-tuning: LoRA proved to be an effective method for fine-tuning LLMs like LLaMA-2 on specialized tasks, balancing performance with computational efficiency.
  • Cloze Test for Extraction: Framing the ADE extraction task as a cloze test showed potential, though it was noted that encoder-only models might have weaker generation capabilities for this approach.
  • Hallucination is a Major Hurdle: The paper highlights that hallucination, particularly "intrinsic hallucination" (errors within the generated text itself), is a significant challenge for LLMs in medical contexts, potentially leading to severe consequences.

Key Results

  • Task 5: The team achieved an impressive F1 score of 0.935 on the test set for Task 5, demonstrating the effectiveness of their LLaMA-2 and LoRA-based approach.
  • Task 1:
    • For extraction, the LLaMA-2 model showed an improvement of around 0.15 in F1 score on the validation set compared to BERT.
  • The overall F1 score for Task 1 was reported as 0.08.
  • The paper notes that for Task 1, approximately 25.9% of the answers suffered from hallucination problems, with intrinsic hallucinations accounting for about 77.2% of these errors.
  • Heuristic rules were applied to correct some intrinsic hallucinations, leading to a 0.015 improvement in F1.

Limitations and Future Work

  • Hallucination: The most significant limitation acknowledged is the issue of LLM hallucination, which poses risks in health-sensitive applications.
  • Task 1 Performance: The reported F1 score for Task 1 is relatively low, indicating room for improvement.
  • Future Work: The authors suggest exploring Retrieval Augmented Generation (RAG) as a potential solution to mitigate hallucination issues. Further research is needed to ensure solid and reliable results from LLMs in the public health domain.

Summary updated

BibTeX
@inproceedings{yu2024polyucbs,
  author = {Zhai Yu and Xiaoyi Bao and Emmanuele Chersoni and Beatrice Portelli and Sophia Lee and Jinghang Gu and Chu-Ren Huang},
  title = {{PolyuCBS at SMM4H 2024: LLM-based Medical Disorder and Adverse Drug Event Detection with Low-rank Adaptation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure