TL;DR
This paper details the WisPerMed team's approach to creating lay summaries of scientific articles for the BioLaySumm 2024 Shared Task. They fine-tuned large language models (LLMs) like BioMistral and Llama3, explored prompt engineering, and developed a Dynamic Expert Selection (DES) mechanism to improve summary quality, ultimately achieving 4th place among 54 participants.
What Problem Does This Paper Address?
Scientific research is crucial for progress, but its findings are often locked away in complex, jargon-filled articles inaccessible to the general public or even researchers in adjacent fields. This creates a barrier to understanding and public engagement. Lay summarization aims to bridge this gap by translating dense scientific text into easily understandable summaries, often requiring additional context beyond the original article. The BioLaySumm 2024 Shared Task specifically challenges participants to develop automatic lay summarization systems for biomedical articles.
Methods
The WisPerMed team employed a multi-pronged strategy to tackle lay summarization:
- Fine-tuning LLMs: They fine-tuned two domain-specific LLMs, BioMistral-7B-DARE (BioM) and Llama3-70B-Instruct (Llama3), using the QLoRA technique. BioM was fine-tuned on abstracts and lay summaries, while Llama3 was trained on entire articles and lay summaries.
- Prompt Engineering: They explored various prompt variations to guide LLM output without fine-tuning. This included:
- An initial prompt for zero-shot and few-shot learning.
- A persona prompt (BioMpers) that instructed the model to act as a science communicator.
- An intro prompt (BioMintro) that included the article's introduction for additional context.
- A guide prompt (BioMguide) that provided explicit instructions on crafting a lay summary.
- Dynamic Expert Selection (DES): This novel mechanism was developed to select the best summary from a set of candidates generated by different prompts. DES uses readability (FKGL, DCRS, CLI) and factuality (AlignScore, SummaC) metrics, normalizing and weighting them to rank and choose the most suitable output. This approach aims to improve summary quality without relying on a reference summary.
Datasets
The experiments utilized the BioLaySumm 2024 Shared Task dataset, which comprises two collections of scientific journal articles and their corresponding lay summaries:
- PLOS: Contains 24,773 training examples and 1,376 validation examples. PLOS lay summaries are approximately 150-200 words and written by authors.
- eLife: Contains 4,346 training examples and 241 validation examples. eLife lay summaries are longer, written by expert editors.
Both datasets include section headings and keywords. A test split of 142 examples was used for evaluation.
Key Insights and Ideas
- Fine-tuning reigns supreme: The experiments strongly suggest that fine-tuning LLMs on domain-specific data (like BioMistral) generally leads to superior performance across most metrics compared to zero-shot or few-shot learning with general-purpose models.
- Few-shot learning is powerful, with the right prompt: While fine-tuning was best, few-shot learning significantly improved models' ability to generate relevant and factually accurate summaries, especially when paired with a well-crafted prompt.
- Context matters, but not always: Including additional context like the introduction (BioMintro) or explicit guidelines (BioMguide) didn't always improve performance and sometimes even led to a decrease in certain metrics. The initial prompt often performed best in few-shot settings.
- Domain-specific LLMs outperform larger general models: BioMistral, a domain-adapted LLM, outperformed the larger Llama3 model in many aspects, highlighting the importance of domain specialization for tasks like biomedical lay summarization.
- DES offers a robust selection mechanism: The Dynamic Expert Selection mechanism proved effective in improving factuality and readability by intelligently selecting the best summary from multiple candidates, even outperforming baseline approaches.
Key Results
- The WisPerMed team achieved 4th place out of 54 participants in the BioLaySumm 2024 Shared Task, measured by readability, factuality, and relevance.
- Their approach improved upon the baseline by approximately 5.5 percentage points in the overall score and was only 1.5 percentage points behind the first-place finisher.
- Fine-tuning BioM yielded the best results in the fine-tuning category, achieving the highest scores in relevance and overall quality.
- In the few-shot setting, the initial prompt for BioM led to the highest readability and factuality, excluding the DES approach.
- The DES mechanism improved all metrics (except a slight drop in LENS) when applied to BioM in the few-shot setting, demonstrating its effectiveness.
- Llama3, despite being a larger model, underperformed, producing lower-quality text and decreased factuality scores.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited prompt exploration: Only four discrete prompts were tested with DES, and only two sets of inference parameters were explored, potentially limiting the full capture of model variability.
- Heuristic DES weights: The weights for the DES method were chosen based on heuristics without formal optimization, which could impact its robustness.
- Potential for prior exposure: There's a possibility that BioM might have been previously exposed to the gold standard summaries, potentially inflating its performance.
Future work could involve:
- More comprehensive prompt engineering and optimization of DES weights.
- Exploring a wider range of tasks and domains to ensure the approach's robustness.
- Adapting these methods for other complex domains or languages.
- Investigating additional metrics for evaluation.
Summary updated