SINAI at BioLaySumm: Self-Play Fine-Tuning of Large Language Models for Biomedical Lay Summarisation

Mariia Chizhikova, Manuel Carlos Díaz-Galiano, L. Alfonso Ureña-López, María-Teresa Martín-ValdiviaACL 2024

TL;DR

This paper tackles the challenge of making complex biomedical research accessible to the public by automatically generating "lay summaries." The SINAI team explored various large language model (LLM) fine-tuning techniques, with their "Self-Play Fine-Tuning" (SPIN) method achieving the best performance in generating relevant and accurate summaries.

What Problem Does This Paper Address?

Scientific advancements are crucial for societal progress, but the technical language used in research papers creates a significant barrier for the general public, journalists, and even researchers in adjacent fields. This makes it difficult for the public to understand and support scientific endeavors, which are often publicly funded. While manual generation of "lay summaries" (plain language explanations) is a solution, it's a time-consuming and expensive process. This paper addresses the need for effective automatic lay summarization systems for biomedical articles, a task highlighted by the BioLaySumm shared task.

Methods

The SINAI team investigated three main approaches for generating lay summaries:

  • Few-Shot Learning (FSL): This involved using pre-trained LLMs (GPT-3.5, GPT-4, and Llama-3) with a small number of examples (2-shot prompting) to guide them in generating summaries. They found that while GPT models produced more complete outputs, Llama-3 required post-processing to remove noise and repetitions.
  • Parameter-Efficient Fine-Tuning: They employed QLoRA (Quantized Low-Rank Adaptation) to fine-tune the Llama-3-8B model. This method is memory-efficient and reduces the number of trainable parameters by backpropagating gradients through a frozen, quantized LLM. They trained a single model on a merged dataset of both PLOS and eLife.
  • Self-Play Fine-Tuning (SPIN): This novel method builds upon a supervised fine-tuned model. The model is trained to generate responses and then learns to distinguish its own generations from the previous step from the gold-standard summaries. This allows the model to improve by essentially "playing against itself" and learning from its own outputs in relation to the ground truth.

Datasets

The study utilized two datasets provided for the BioLaySumm shared task:

  • PLOS (Public Library of Science): A large-scale dataset with 24,773 training instances and 1,376 validation instances. PLOS articles often include author summaries of 150-200 words.
  • eLife: A medium-scale dataset with 4,246 training instances and 241 validation instances. eLife has been creating plain-language summaries since 2021.

A key difference noted is that eLife lay summaries tend to be longer and more abstractive than PLOS summaries.

Key Insights and Ideas

  • SPIN's Effectiveness: The Self-Play Fine-Tuning (SPIN) method emerged as the most promising approach, demonstrating superior performance in generating relevant lay summaries. This suggests that models can learn effectively by iteratively refining their outputs against known good examples.
  • Trade-off Between Readability and Factuality: The study observed a clear trade-off between how readable a summary is and how factually accurate it is. Models that produced highly readable summaries sometimes sacrificed factual precision, and vice-versa.
  • GPT vs. Llama-3 Performance: While GPT models (3.5 and 4) showed more robust initial outputs in FSL, Llama-3, especially when fine-tuned with SPIN, achieved competitive and in some cases better results, highlighting the potential of open-source models.
  • Post-processing is Crucial: For FSL with Llama-3, rule-based post-processing was essential to clean up noisy or repetitive outputs, demonstrating the importance of integrating such steps for practical LLM applications.

Key Results

The SPIN fine-tuning method achieved the highest scores among the SINAI team's submissions, with:

  • ROUGE-1: 0.4205
  • BERTScore: 0.8583

In terms of readability and factuality, the fine-tuned Llama-3-8B with QLoRA achieved the best factuality scores while maintaining good readability. Conversely, GPT-4 FSL systems produced more complex text but had lower factuality scores.

Limitations and Future Work

The authors acknowledge that the pre-trained LLMs used in their study may inherit biases from their training data, which could lead to inappropriate content or the replication of existing biases. They emphasize the need for comprehensive safety and fairness evaluations before deploying these systems in real-world applications. Future work could explore further advancements in self-play mechanisms and more robust methods for ensuring both accuracy and accessibility in lay summaries.

Summary updated

BibTeX
@inproceedings{chizhikova2024sinai,
  author = {Mariia Chizhikova and Manuel Carlos Díaz-Galiano and L. Alfonso Ureña-López and María-Teresa Martín-Valdivia},
  title = {{SINAI at BioLaySumm: Self-Play Fine-Tuning of Large Language Models for Biomedical Lay Summarisation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure