Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study

Zhaoyue Sun, Gabriele Pergola, Byron C. Wallace, Yulan He

Abstract

Pharmacovigilance (PV) is critical for drug safety, requiring the precise extraction of complex, hierarchical medication events from unstructured medical text. This study evaluates the efficacy of Large Language Models (LLMs), specifically ChatGPT (gpt-3.5-turbo-0301), in performing Pharmacovigilance Event Extraction (PHEE). We investigate various prompt engineering strategies, in-context learning techniques, and the potential of LLMs as data synthesizers for training smaller, specialized models. Our findings indicate that while ChatGPT shows promise in few-shot scenarios, it remains outperformed by smaller, fully fine-tuned models like Flan-T5. Furthermore, we demonstrate that synthetic data generated by LLMs requires rigorous filtering to prevent performance degradation in downstream tasks.

Introduction

Pharmacovigilance involves the detection, assessment, and prevention of adverse drug effects. Historically, this process has relied on manual review of electronic health records (EHRs) and medical case reports—a labor-intensive and error-prone task. Automating this extraction is a significant challenge in natural language processing (NLP) due to the hierarchical nature of medical events. An extraction task must identify not just a drug name, but a structured event including the patient, dosage, and specific adverse effects.

While LLMs have demonstrated remarkable general-purpose capabilities, their performance in high-stakes, linguistically dense medical domains remains a subject of active research. This study explores whether ChatGPT can bridge the gap in PHEE and whether it can serve as a "data synthesizer" to improve the training of smaller, more efficient models.

Methodology

Prompt Engineering and In-Context Learning

The researchers evaluated ChatGPT using four distinct prompting strategies:

  1. Schema: Providing a simple list of event and argument types.
  2. Explanation: Including detailed natural language definitions for every argument (e.g., distinguishing between "subject" and "population").
  3. Code: Structuring the prompt as a Python-like class definition.
  4. Pipeline: A multi-step approach where the model first identifies main arguments and subsequently answers follow-up questions to extract sub-arguments.

For few-shot learning, the study compared different methods for selecting the most relevant examples: random selection, semantic similarity (SBERT), lexical similarity (BM25), and structural similarity (TreeKernel).

Data Augmentation and Filtering

To test ChatGPT's utility as a data generator, the researchers used the model to synthesize medical sentences. Recognizing that LLM-generated labels can be "noisy" or inaccurate, they implemented a filtering strategy. A secondary fine-tuned model was used to calculate the probability of the generated sequences; examples that caused high "confusion" in the secondary model were discarded to maintain training data quality.

The PHEE Dataset

The study utilized the PHEE dataset, a specialized benchmark consisting of approximately 5,000 sentences from public medical reports. The dataset focuses on two primary event types:

  • Adverse Drug Events (ADE): Harmful effects resulting from medication.
  • Potential Therapeutic Events (PTE): Beneficial or intended effects of a drug.

Key Results and Analysis

Performance Benchmarks

The study revealed a significant performance gap between general-purpose LLMs and specialized models. A fully fine-tuned Flan-T5 (Large) model achieved the state-of-the-art result with an Exact Match F1 (EM_F1) score of ~70.8 for main arguments. In contrast, ChatGPT’s best few-shot performance (utilizing BM25 for example selection) reached an EM_F1 of 60.39. In zero-shot settings, ChatGPT struggled significantly, achieving only 34.80 EM_F1.

Prompting and Selection Insights

  • Lexical vs. Semantic Similarity: BM25 (lexical similarity) outperformed SBERT (semantic similarity) in few-shot selection. This suggests that in medical event extraction, the presence of specific entities like drug names is a stronger indicator of relevance than broader semantic meaning.
  • The Power of Explanation: In zero-shot scenarios, providing detailed natural language explanations of the schema provided a substantial performance boost compared to simple label lists.

The Risks of Synthetic Data

Directly incorporating ChatGPT-generated data into the training set for smaller models actually decreased performance. The synthetic data often lacked coverage for rare arguments and struggled with complex entity boundaries. However, the introduction of a "judge" model to filter low-confidence synthetic examples helped stabilize the training process, though consistent improvement via augmentation remains difficult to achieve.

Discussion and Limitations

The analysis identified several persistent challenges for ChatGPT in the medical domain:

  1. Semantic Confusion: The model frequently confused "subject disorder" (the patient's pre-existing condition) with "treatment disorder" (the indication for the drug).
  2. Rare Argument Bias: Synthetic data tended to focus on common patterns, leading to a "rich get richer" bias that neglected rare but critical arguments.
  3. Efficiency: The "Pipeline" prompting method, while thorough, was 10 times slower and significantly more expensive than end-to-end generation.

Conclusion

While ChatGPT is a powerful tool for medical NLP, it does not yet replace specialized, fine-tuned models in complex tasks like Pharmacovigilance Event Extraction. The 10-point F1 gap between ChatGPT and Flan-T5 highlights the necessity of domain-specific fine-tuning. Future research should explore Chain-of-Thought (CoT) reasoning and the use of larger open-source models to balance performance with the data privacy requirements inherent in healthcare.

Summary updated

BibTeX
@inproceedings{sun2024leveraging,
  author = {Zhaoyue Sun and Gabriele Pergola and Byron C. Wallace and Yulan He},
  title = {{Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study}},
  booktitle = {EACL},
  year = {2024},
  url = {https://openreview.net/forum?id=BLB8XU08Hm},
}
Figure