Improving Entity Recognition Using Ensembles of Deep Learning and Fine-tuned Large Language Models: A Case Study on Adverse Event Extraction from Multiple Sources

Yiming Li, Deepthi Viswaroopan, William He, Jianfu Li, Xu Zuo, Hua Xu, Cui TaoarXiv 2024 · DOI · arXiv

TL;DR

Researchers have developed a high-performing ensemble method that combines fine-tuned Large Language Models (LLMs) with traditional deep learning architectures to extract vaccine adverse events from social media and clinical reports. By leveraging the contextual "intuition" of models like GPT-3.5 alongside the structural reliability of BioBERT and RNNs, the team achieved a 90.3% strict F1-score, significantly outperforming individual models.

What Problem Does This Paper Address?

Monitoring the safety of vaccines—pharmacovigilance—is a monumental task, especially during a global pandemic. While official systems like the Vaccine Adverse Event Reporting System (VAERS) are vital, they are often supplemented by "real-world" data from social media (Twitter, Reddit), where patients describe side effects in informal, noisy language.

The challenge lies in Named Entity Recognition (NER): automatically identifying specific terms like the vaccine name, the dose number (shot), and the adverse event (AE) itself. Traditional deep learning models (like RNNs) are great at spotting patterns but require massive amounts of labeled data. Modern LLMs (like GPT-4) have incredible contextual understanding but can be "unstable" or imprecise when tasked with niche medical extraction. This paper asks: Can we combine these two different "ways of thinking" to create a more robust extraction tool?

Methods

The researchers didn't just pick one winner; they built a "committee of experts." Their approach involved:

  1. Fine-tuning: They took several models—including GPT-2, GPT-3.5, Llama-2 (7b and 13b), BioBERT, and a Recurrent Neural Network (RNN)—and fine-tuned them specifically on a labeled dataset of vaccine reports.
  2. Prompt Engineering: For the LLMs, they tested two styles: "Split" prompts (asking for one entity type at a time) and "Merged" prompts (asking for everything at once).
  3. The Ensemble: The "secret sauce" was a majority voting ensemble. They took the three top-performing individual models—Fine-tuned GPT-3.5, BioBERT, and the RNN—and had them vote on every entity. If two or more models agreed on a label, that became the final output.

Datasets

The study utilized a diverse corpus of 9,885 total documents collected between December 2020 and December 2022:

  • VAERS: 621 official reports.
  • Twitter: 9,133 tweets (filtered to ensure they described personal experiences rather than just news).
  • Reddit: 131 posts from various subreddits.

The researchers manually annotated these for three entities: vaccine (e.g., "Pfizer"), shot (e.g., "second dose"), and ae (e.g., "migraine").

Key Insights and Ideas

  • Fine-Tuning is Mandatory: One of the most striking findings was that even "smart" models like GPT-4 struggled in a zero-shot (un-tuned) setting compared to much smaller models that were fine-tuned on the specific task.
  • Complementary Strengths: LLMs excel at capturing complex linguistic nuances and "reading between the lines" in messy social media posts. Traditional models like BioBERT provide a "structural anchor," ensuring that entity boundaries are identified with high precision.
  • The Power of the Ensemble: Combining these models led to an 8% jump in strict F1-score. This suggests that the errors made by LLMs and traditional models are often different; where one trips up, the other compensates.
  • The "Shot" Paradox: Interestingly, the shot entity (identifying which dose was taken) had high inter-rater agreement among humans but was rare in the data, making it a difficult "low-frequency" challenge for the models to learn.

Key Results

The ensemble approach was the clear victor across the board:

  • Overall Performance: Achieved a 0.903 micro-average strict F1-score.
  • Entity Specifics (Strict F1):
    • vaccine: 0.878
    • shot: 0.930
    • ae: 0.925
  • Error Analysis: The models were remarkably accurate at identifying the type of entity (0% incorrect entity type errors). Most mistakes were "false negatives" for adverse events, where the model missed subtle descriptions like "a knot in my hairline" or "achiness."

Limitations and Future Work

While the results are promising, the authors note a few hurdles:

  • Data Bias: Social media data is inherently biased toward certain demographics and may not represent the entire population's experience with vaccines.
  • Ontology Gaps: Some missed adverse events were highly specific or colloquial terms not well-represented in the training data or standard medical ontologies.
  • Generalizability: The study focused strictly on COVID-19 vaccines. Future work is needed to see if this ensemble framework holds up for other medical domains, such as oncology or chronic disease management.

The researchers plan to continue refining LLM prompting strategies and expanding their datasets to capture an even wider variety of patient-reported symptoms.

Summary updated

BibTeX
@misc{li2024improving,
  author = {Yiming Li and Deepthi Viswaroopan and William He and Jianfu Li and Xu Zuo and Hua Xu and Cui Tao},
  title = {{Improving Entity Recognition Using Ensembles of Deep Learning and Fine-tuned Large Language Models: A Case Study on Adverse Event Extraction from Multiple Sources}},
  year = {2024},
  eprint = {2406.18049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.18049},
}
Figure