TL;DR
This paper explores using pre-trained large language models (LLMs) like BART and LED to automatically generate lay summaries of complex biomedical research articles. The authors successfully fine-tuned these models, achieving high relevance and factuality scores and ranking among the top five systems in the BioLaySumm 2023 challenge.
What Problem Does This Paper Address?
Biomedical research articles are dense with technical jargon, making them inaccessible to the general public, including patients and their families. This creates a knowledge gap, hindering informed decision-making about health. The paper addresses the challenge of automatically generating concise, readable summaries of these articles in a "layperson-friendly" language, bridging the communication gap between scientific experts and the broader audience.
Methods
The researchers employed two prominent sequence-to-sequence (seq2seq) LLMs: BART (Bidirectional and Auto-Regressive Transformer) and LED (Longformer Encoder-Decoder).
- BART: They utilized the
facebook/bart-large-xsumvariant, pre-trained for summarization. To overcome memory constraints on their GPU, they implemented mixed-precision training (fp16), gradient accumulation, and gradient checkpointing. A key limitation of BART is its fixed input token limit (1024 tokens), meaning longer articles were truncated. - LED: To address BART's token limitation, they adopted the
allenai/led-large-16384-arxivvariant, which can handle significantly longer sequences (up to 8192 tokens). Similar memory optimization techniques were applied, though the batch size had to be reduced to 1 due to hardware limitations.
The models were fine-tuned on provided datasets for two tasks: generating lay summaries (Task 1) and generating both lay summaries and technical abstracts (Task 2). For Task 2, they experimented with both using a pre-trained model and retraining it specifically on the abstract sections of articles.
Datasets
The study utilized two datasets for the BioLaySumm 2023 shared tasks:
- PLOS Dataset: This dataset comprised 24,773 articles for training and 1,376 for validation. The median article length was 6,577 words. This dataset was used for both Task 1 and Task 2.
- eLife Dataset: This dataset contained 4,346 articles for training and 241 for validation. The articles in this dataset were longer, with a median length of 9,837.5 words. This dataset was used for Task 1.
Key Insights and Ideas
- LLMs for Lay Summarization are Promising: The research demonstrates that pre-trained LLMs, when appropriately fine-tuned, can generate high-quality lay summaries of complex biomedical texts, significantly improving accessibility.
- Longformer's Advantage for Long Documents: LED's ability to process much longer sequences than BART proved beneficial, although the full potential might not have been realized due to training constraints.
- Retraining for Specific Tasks is Crucial: For generating technical abstracts (Task 2), simply fine-tuning a model trained for lay summaries wasn't sufficient. Retraining the model specifically on the abstract sections of articles significantly improved performance, highlighting the importance of task-specific data.
- Trade-offs in Model Selection: While BART is more computationally efficient, LED's longer context window is essential for truly capturing the essence of lengthy research papers. However, hardware limitations can constrain the effective use of models like LED.
Key Results
- Task 1 (Lay Summarization): The final system, using a combination of BART and LED, achieved strong performance. It was ranked the best system on relevance and the second best on factuality. Overall, it secured the fifth position among 20 participants.
- Task 2 (Technical Abstract Generation): The initial approach for technical abstracts was less successful. However, after retraining the BART model on PLOS abstracts, the performance improved significantly, leading to a joint first-place ranking for this task.
- Readability: While relevance and factuality were strong points, readability was noted as an area where the model's performance was not as high, suggesting room for improvement.
Limitations and Future Work
- Computational Resources: The authors acknowledge that LLMs demand significant time and memory resources, which limited their ability to explore more extensive hyperparameter tuning or longer training epochs.
- Readability Improvement: The models' readability scores were not as high as desired. Future work could involve developing methods to identify and substitute technical jargon with simpler synonyms, potentially using machine learning or list-based approaches.
- Bias in LLMs: The paper highlights the ethical consideration that LLMs can inherit biases from their training data, which could lead to underrepresentation or misrepresentation of certain groups in summaries. Awareness and mitigation of these biases are crucial.
- Accuracy and Reliability: LLMs are not always perfectly accurate or reliable, and inaccuracies in generated summaries could have serious health implications. Ensuring the robustness and accuracy of these models remains an ongoing challenge.
Summary updated