TL;DR
This paper tackles the challenge of creating lay summaries of complex biomedical research for a general audience. The authors investigate how pre-trained language models, specifically FLAN-T5, perform in this task, exploring the impact of model size and training data. Their approach achieved a strong second place in the BioLaySumm shared task, highlighting the effectiveness of generative models for this specialized summarization.
What Problem Does This Paper Address?
Lay summarization, the task of generating summaries understandable to a non-expert audience, is crucial for disseminating scientific knowledge. However, it remains an under-explored area, especially for highly technical fields like biomedicine. Existing summarization models often struggle to simplify complex terminology and maintain readability without sacrificing accuracy. This paper addresses this gap by systematically evaluating pre-trained generative language models for their ability to produce effective lay summaries of biomedical research articles, focusing on the critical factors of model scale and data augmentation.
Methods
The core of the approach involves fine-tuning pre-trained language models, specifically FLAN-T5, on lay summarization tasks. The authors experimented with different sizes of FLAN-T5 (base, large, and XL) to assess the impact of model scale. To address the limited size of the eLife training dataset, they explored intermediate-task pre-training, fine-tuning models on related datasets (like PLOS or XSum) before the target task. A novel aspect is the use of GPT-3.5 to generate synthetic training data, either by paraphrasing existing summaries or by generating summaries from scratch for documents. This data augmentation strategy was investigated to improve model performance, particularly on smaller datasets. For inference, a beam search decoder with a beam width of four was used to generate summaries up to 512 tokens.
Datasets
The primary datasets used for this research are:
- eLife: A scientific journal for biomedical and life sciences, where publications include lay summaries written by expert editors.
- PLOS (Public Library of Science): An open-access journal focusing on science and medicine, with lay summaries provided by the authors.
Both datasets were used for training and development, with separate test sets provided by the BioLaySumm shared task organizers. The paper also mentions using the XSum dataset for intermediate-task pre-training.
Key Insights and Ideas
- Model Size Matters (for Relevance): Larger FLAN-T5 models generally lead to better relevance scores (ROUGE, BERTScore), indicating they can capture more of the essential information.
- Readability is Tricky: While larger models improve relevance, they don't necessarily improve readability (FKGL, DCRS) or factuality (BARTScore). In some cases, smaller models even achieved better scores on these metrics.
- Data Augmentation is Powerful: Using GPT-3.5 to generate additional training data significantly boosted relevance metrics, even outperforming larger datasets like PLOS or XSum when used for transfer learning. This suggests synthetic data can be a cost-effective way to improve lay summarization.
- Input Length Trade-offs: Truncating input documents negatively impacts relevance. While longer inputs can slightly improve relevance, excessively long inputs (beyond 2000 tokens) lead to performance degradation, likely due to difficulties in capturing long-range dependencies. Readability and factuality metrics are less sensitive to input length.
Key Results
The authors' submission, using FLAN-T5-XL with GPT-3.5 generated data, ranked second for relevance criteria and third overall among 21 competing teams in the BioLaySumm shared task. Specifically, their approach achieved:
- eLife Test Set: ROUGE-1: 0.480, ROUGE-2: 0.130, ROUGE-L: 0.454, BERTScore: 0.854, FKGL: 9.804, DCRS: 8.224, BARTScore: -2.493.
- PLOS Test Set: ROUGE-1: 0.497, ROUGE-2: 0.194, ROUGE-L: 0.460, BERTScore: 0.867, FKGL: 15.089, DCRS: 11.372, BARTScore: -1.887.
The paper also shows that GPT-3.5 paraphrasing (GPT-3.5 (P)) on the eLife test set yielded the best relevance scores, outperforming other data augmentation strategies.
Limitations and Future Work
The authors acknowledge that they did not explicitly analyze the faithfulness and factuality of the GPT-3.5 generated examples. This could lead to the propagation of errors into the generated summaries. Future work could involve using faithfulness and factuality metrics to assess generated training examples and employing post-editing or human evaluation to refine them. Additionally, developing summarization models that specifically optimize for readability and factuality, beyond relevance, is suggested as a future direction.
Summary updated