Eulerian at BioLaySumm: Preprocessing Over Abstract is All You Need

Satyam Modi, T KarthikeyanACL 2024

TL;DR

This paper tackles the challenge of automatically generating easy-to-understand summaries of complex biomedical research articles for a general audience. The authors found that a simple preprocessing step of extracting and cleaning the abstract, combined with a pre-trained language model (Flan-T5), surprisingly outperformed more complex fine-tuning strategies, achieving top scores in factuality.

What Problem Does This Paper Address?

The sheer volume of biomedical research published daily makes it incredibly difficult for anyone outside a specific niche to keep up with the latest findings. This is especially problematic for patients, healthcare providers, and policymakers who need to understand this research but lack the specialized vocabulary and background knowledge. Manual summarization is time-consuming and impractical at scale. Therefore, developing automated systems that can generate accurate and accessible "lay summaries" from dense scientific articles is a crucial goal. The BioLaySumm 2024 Shared Task specifically aims to advance this capability.

Methods

The core of the authors' approach revolves around two main components:

  • Preprocessing Over Abstract (PoA): This is a novel preprocessing technique. It involves extracting the initial sentences of a research paper, which typically constitute the abstract. Crucially, it then applies regular expressions to remove content within parentheses, braces, and brackets, as these often contain supplementary details that can hinder readability for a lay audience. The goal is to distill the core information while improving clarity.
  • Fine-tuning Flan-T5 Models: The researchers utilized Flan-T5, a powerful pre-trained language model known for its strong performance on various text-to-text tasks. They experimented with different versions of Flan-T5 (small and base) and trained them on two datasets (PLOS and eLife). They explored various training strategies, including fine-tuning on individual datasets versus combined datasets, and the impact of a cosine learning rate scheduler.

Datasets

The study utilized two publicly available datasets for the BioLaySumm 2024 task:

  • PLOS: Derived from the Public Library of Science, this dataset contains 24,773 training instances and 1,376 validation instances.
  • eLife: This dataset comes from the peer-reviewed eLife journal and includes 4,346 training instances and 241 validation instances.

The test data for evaluation consisted of 142 articles from each dataset.

Key Insights and Ideas

  • Simplicity Wins (Sometimes): The most striking finding is that the simple "Preprocessing Over Abstract" (PoA) method, which requires no training, outperformed all the fine-tuned Flan-T5 models on key factuality and relevance metrics. This suggests that carefully curated input can be more impactful than extensive model training for this specific task.
  • Abstract is Key: The PoA method's success highlights the inherent value of the abstract in conveying the core message of a research paper. By cleaning and focusing on this section, a surprisingly effective summary can be generated.
  • Trade-offs in Model Size and Training: While larger Flan-T5 models (base vs. small) and more diverse training data (PLOS + eLife) generally improved relevance and readability, they often came at the cost of factuality. This indicates a delicate balance to strike between generating fluent, easy-to-read summaries and ensuring they accurately reflect the original research.
  • Scheduler Matters for Balance: The cosine learning rate scheduler helped the Flan-T5 base model achieve a better balance between readability and factuality, suggesting that fine-tuning strategies can indeed mitigate some of the trade-offs observed with larger models.

Key Results

The paper reports impressive results, particularly with the PoA method:

  • Top Factuality: The authors achieved the highest scores on the "Factuality metric" (composed of AlignScore and SummaC) among all participating teams, with an AlignScore of 0.9914 and a SummaC score of 0.944.
  • PoA Outperforms Models: The PoA method alone achieved an AlignScore of 0.9914 and a SummaC score of 0.944, surpassing the best-performing Flan-T5 model in factuality.
  • Flan-T5 Performance: The Flan-T5 base model, when fine-tuned on combined PLOS and eLife data with a cosine scheduler, achieved good relevance and readability scores, but its factuality metrics were lower than the PoA method. For instance, this model achieved an AlignScore of 0.9122 and SummaC of 0.8385.

Limitations and Future Work

  • Readability of PoA: While PoA excels in factuality and relevance, its readability scores were noted to be lower, likely due to the inherent complexity of abstracts.
  • Dataset Nuances: Combining PLOS and eLife datasets might not capture the unique characteristics of each. Future work could involve training separate models for each dataset.
  • Model Exploration: The authors suggest exploring autoregressive large language models (like LLaMA) and advanced prompt tuning techniques (with models like GPT-4 and Gemini) for further improvements.
  • Broader Datasets and Fine-tuning: Augmenting training datasets to cover a wider range of biomedical texts and exploring advanced fine-tuning techniques like mixed precision and curriculum learning are also proposed.
  • Domain-Specific Models: Tailoring models for specific sub-domains within biomedical research could enhance accuracy and relevance.
  • Comprehensive Evaluation: Developing evaluation frameworks that incorporate user satisfaction and practical utility alongside traditional metrics is crucial.

Summary updated

BibTeX
@inproceedings{modi2024eulerian,
  author = {Satyam Modi and T Karthikeyan},
  title = {{Eulerian at BioLaySumm: Preprocessing Over Abstract is All You Need}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure