Closing the gap between open source and commercial large language models for medical evidence summarization

Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas Campion, Zhiyong Lu, Chunhua Weng, Yifan PengNpj Digit. Medicine 2024 · OpenReview

TL;DR

This paper tackles the performance gap between proprietary and open-source Large Language Models (LLMs) for summarizing medical evidence. By fine-tuning open-source models like PRIMERA, LongT5, and Llama-2 on a new dataset, they significantly improved their summarization quality, making them competitive with commercial models and even outperforming larger zero-shot models.

What Problem Does This Paper Address?

Systematic reviews and meta-analyses are crucial for evidence-based medicine, but their creation is time-consuming and reviews quickly become outdated. Automating the summarization of medical evidence is therefore highly desirable. While commercial LLMs show promise, they come with risks like lack of transparency and vendor lock-in. Open-source LLMs offer transparency and customization but have historically lagged in performance for specialized tasks like medical summarization. This paper investigates whether fine-tuning these open-source models can bridge this performance gap.

Methods

The core of this research lies in fine-tuning open-source LLMs. Instead of relying on zero-shot prompting (where models generate output based on their pre-training without specific task examples), the researchers adapted existing LLMs to the medical summarization task. They employed Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning technique. LoRA allows for significant performance gains by updating only a small subset of model parameters, making the fine-tuning process more computationally feasible and reducing the risk of "catastrophic forgetting" (where a model loses its general capabilities). They fine-tuned three specific open-source LLMs: PRIMERA, LongT5, and Llama-2.

Datasets

The primary dataset used in this study is MedReview, a newly introduced benchmark consisting of 8,161 pairs of systematic review abstracts and their corresponding narrative summaries. This dataset is an extension of previous work and covers a wide range of medical topics. The data was split into training, validation, and test sets, with the test set specifically curated to include articles published after the knowledge cutoff dates of most LLMs to ensure a fair evaluation.

Key Insights and Ideas

  • Fine-tuning is key: The study strongly demonstrates that fine-tuning is a powerful strategy to significantly boost the performance of open-source LLMs for specialized tasks, effectively closing the gap with proprietary models.
  • Smaller can be better (when fine-tuned): Surprisingly, smaller fine-tuned models sometimes outperformed larger, zero-shot commercial models, highlighting the efficiency and effectiveness of domain-specific adaptation.
  • Open-source is viable for critical tasks: The research shows that open-source LLMs, when properly fine-tuned, can achieve performance levels suitable for demanding applications like medical evidence summarization, offering a transparent and customizable alternative.
  • Beyond lexical similarity: The study emphasizes the limitations of traditional automatic metrics (like ROUGE) and introduces PICO extraction as a more robust way to evaluate factual completeness and consistency in medical summaries.

Key Results

  • Fine-tuned LLMs showed substantial improvements across multiple metrics: an average increase of 9.89% in ROUGE-L, 13.21% in METEOR, and 15.82% in CHRF.
  • The fine-tuned LongT5 model achieved performance comparable to zero-shot GPT-3.5-turbo.
  • In human evaluations, fine-tuned models achieved win-rates of 59.20% for Llama-2, 54.47% for PRIMERA, and 59.68% for LongT5 against zero-shot Llama-2.
  • GPT-4 simulated evaluations also confirmed significant improvements after fine-tuning, with 77.97% accuracy for fine-tuned LongT5 on "after cutoff" data.
  • Fine-tuned models demonstrated better comprehensiveness, consistency, readability, and specificity compared to their zero-shot counterparts.

Limitations and Future Work

The authors acknowledge that due to resource constraints, they could not extensively explore all proprietary LLMs (like Claude, GPT-3.5-turbo, and GPT-4) in their fine-tuning experiments. Their study focused on open-source models. A future direction suggested is to deploy LLMs to directly synthesize information from clinical trial publications, rather than just review abstracts. They also note that while fine-tuning improves summarization, it doesn't guarantee absolute truthfulness or accuracy, and trustworthy summarization remains a challenging, unresolved issue.

Summary updated

Abstract

Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to the proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance. Utilizing a benchmark dataset, MedReview, consisting of 8161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the performance of open-source models was all improved after fine-tuning. The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were manifested in both a human evaluation and a larger-scale GPT4-simulated evaluation.
BibTeX
@inproceedings{zhang2024closing,
  author = {Gongbo Zhang and Qiao Jin and Yiliang Zhou and Song Wang and Betina Idnay and Yiming Luo and Elizabeth Park and Jordan G. Nestor and Matthew E. Spotnitz and Ali Soroush and Thomas Campion and Zhiyong Lu and Chunhua Weng and Yifan Peng},
  title = {{Closing the gap between open source and commercial large language models for medical evidence summarization}},
  booktitle = {Npj Digit. Medicine},
  year = {2024},
  url = {https://openreview.net/forum?id=W5DbAoFnsy},
}
Figure