DeakinNLP at BioLaySumm: Evaluating Fine-tuning Longformer and GPT-4 Prompting for Biomedical Lay Summarization

Huy Quoc To, Ming Liu, Guangyan HuangACL 2024

TL;DR

This paper explores two methods for generating "lay summaries" of biomedical articles: fine-tuning the Longformer-Encoder-Decoder (LED) model and using GPT-4 with zero-shot and few-shot prompting. They found that while fine-tuned LED performs better on relevance and factuality, GPT-4 excels in readability and is a better option when training data is scarce.

What Problem Does This Paper Address?

Biomedical research articles are dense and technical, making them inaccessible to the general public. The BioLaySumm 2024 shared task aims to address this by generating "lay summaries" – simplified, understandable versions of these articles. The challenge lies in creating summaries that are not only accurate and relevant but also easy to read. This paper investigates how well two distinct approaches, a specialized fine-tuned model (LED) and a powerful general-purpose large language model (GPT-4), can tackle this problem, especially considering the long nature of biomedical texts.

Methods

The researchers employed two primary methods:

  1. Fine-tuning Longformer-Encoder-Decoder (LED): They took the pre-trained LED model, known for its ability to handle long documents, and fine-tuned it on two biomedical datasets (PLOS and eLife). This fine-tuning was performed using either 50% or 100% of the available training data. The goal was to adapt LED specifically for the lay summarization task.
  2. GPT-4 Prompting: They utilized GPT-4, a state-of-the-art large language model, in two prompting strategies:
  3. Zero-shot prompting: GPT-4 was given the research article's abstract and conclusion and asked to generate a lay summary directly, without any prior examples.
  4. Few-shot prompting: GPT-4 was provided with a few examples of research articles and their corresponding lay summaries before being asked to summarize a new article. This leverages GPT-4's in-context learning capabilities.

Datasets

The study used two publicly available datasets for biomedical lay summarization:

  • PLOS: Contains biomedical articles and manually written lay summaries.
  • eLife: Similar to PLOS, it includes biomedical articles and their corresponding lay summaries.

The paper notes that eLife articles tend to be longer than those in PLOS.

Key Insights and Ideas

  • Abstracts are Key: A preliminary study revealed that the abstracts and conclusions of biomedical articles contain the most relevant information for generating lay summaries, suggesting these sections can serve as effective input for summarization models.
  • Fine-tuning vs. Prompting Trade-offs: While fine-tuning LED yielded strong results in relevance and factuality, GPT-4 demonstrated superior performance in readability (simplicity and ease of understanding).
  • Data Scarcity Favors GPT-4: In situations with limited training data (like the eLife dataset, which has fewer instances than PLOS), GPT-4's few-shot prompting approach proved more effective than fine-tuning LED.
  • Readability is a Challenge for Fine-tuning: Fine-tuned LED models struggled with readability metrics like LENS, FKGL, DCRS, and CLI, indicating that while they can capture factual content, they may not simplify it as effectively as GPT-4.

Key Results

  • PLOS Dataset: Fine-tuning LED with 100% of the training data achieved the best overall performance on relevance and factuality metrics. However, GPT-4 outperformed LED in readability scores.
  • eLife Dataset: GPT-4's few-shot prompting significantly outperformed the fine-tuned LED model in generating more accurate summaries. GPT-4 also achieved better readability scores on this dataset.
  • Final Submission: The authors' final submission combined the fine-tuned LED model (100% training data) for the PLOS dataset and GPT-4 few-shot prompting for the eLife dataset. This submission surpassed a BART baseline on several metrics, particularly LENS.
  • Cost vs. Performance: While GPT-4 prompting incurs querying costs, it requires no training time, making it a faster alternative. Fine-tuning LED, especially with more data, involves significant training time and computational cost.

Limitations and Future Work

  • API Dependency: The reliance on OpenAI APIs for GPT-4 limits the scope of experimentation due to cost and the inability to reproduce results precisely without access to the exact model version.
  • Limited Fine-tuning: Due to computational constraints, the LED model was not fine-tuned for an extensive number of epochs, potentially impacting its ultimate performance.
  • Future Directions: The authors suggest exploring parameter-efficient fine-tuning techniques and self-evaluation methods for future work to address cost and efficiency concerns. They also propose investigating how to improve the readability of fine-tuned models.

Summary updated

BibTeX
@inproceedings{to2024deakinnlp,
  author = {Huy Quoc To and Ming Liu and Guangyan Huang},
  title = {{DeakinNLP at BioLaySumm: Evaluating Fine-tuning Longformer and GPT-4 Prompting for Biomedical Lay Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure