Towards Knowledge-Guided Biomedical Lay Summarization using Large Language Models

Shufan Ming, Yue Guo, Halil KilicogluNAACL 2025

TL;DR

This paper tackles the challenge of making complex biomedical research accessible to the public by using Large Language Models (LLMs) for lay summarization. They introduce a novel multi-turn dialogue approach that leverages MeSH terms to guide LLMs in generating more relevant background information, significantly improving summary quality and readability.

What Problem Does This Paper Address?

The sheer volume and technical jargon in biomedical literature make it incredibly difficult for laypeople to stay informed about the latest scientific advancements. While automatic summarization aims to condense information, lay summarization requires an extra layer of complexity: incorporating external background knowledge and explaining technical terms that aren't explicitly defined in the source article. This gap in accessibility hinders public understanding of crucial health research.

Methods

The core of this work is a multi-turn instruction tuning approach for LLMs. Instead of a single prompt, the model engages in a two-turn conversation:

  1. MeSH Prediction (Auxiliary Task): The first turn focuses on predicting relevant Medical Subject Headings (MeSH) terms that describe the article's main topics. This is done using the article's abstract as input.
  2. Lay Summary Generation: The second turn uses the predicted MeSH terms as external guidance to generate a lay summary, conditioned on both the original abstract and the MeSH terms.

A key innovation is the heuristic-based curation of MeSH terms. To avoid overly general terms that could mislead the LLM, they filter MeSH terms based on their hierarchical structure, prioritizing more specific terms. This ensures the guidance is focused and relevant. They also explore different training objectives, including a single-turn approach and an "instruction modeling" approach that trains on both user input and assistant responses.

Datasets

The primary dataset used is the eLife dataset, which contains 4,346 pairs of full-text articles and lay summaries for training, along with validation and test sets. This dataset is noted for its longer, expert-written lay summaries and a strong content bias towards research background, making it a good benchmark for this task.

Additionally, a held-out dataset of 71 eLife articles published after the LLaMA model's knowledge cutoff date was collected to evaluate the model's generalizability to unseen data.

Key Insights and Ideas

  • MeSH terms are powerful guides: Incorporating MeSH terms, even as a simple prompt augmentation, significantly improves summary relevance and readability. They act as a crucial "proxy" for guiding LLMs towards coherent, relevant, and understandable lay summaries.
  • Multi-turn dialogue enhances reasoning: The two-turn approach, where MeSH prediction informs summary generation, allows the LLM to engage in a step-by-step reasoning process, mimicking chain-of-thought, which leads to better summaries.
  • Specificity matters in MeSH guidance: Overly general MeSH terms can be detrimental. A curated, hierarchical selection of MeSH terms is more effective than using the full list or relying on less specific terms.
  • Generalizability is key: The multi-turn approach with MeSH generation shows better generalization to unseen data compared to single-turn methods, suggesting it's less prone to memorization.

Key Results

  • The best model, INSTRUCTION_TUNING (multi-turn with curated MeSH), improved the state-of-the-art on the eLife test set, achieving a nearly 2% increase in ROUGE-1 score compared to prior work.
  • This model also showed statistically significant improvements in ROUGE-2 and all readability metrics compared to a baseline without MeSH.
  • The held-out dataset evaluation demonstrated that the INSTRUCTION_TUNING model maintains strong performance, indicating good generalizability.
  • In-context learning with MeSH guidance (MESH_GUIDANCE) also showed significant improvements in relevance metrics over a basic prompt, though readability scores decreased.

Limitations and Future Work

  • Computational Costs: The study was limited by computational resources, leading to the use of only abstracts as input and testing on a single dataset.
  • Faithfulness Evaluation: The authors acknowledge the difficulty in evaluating summary faithfulness with automatic metrics that align well with human judgment, especially for abstractive summarization that requires external knowledge.
  • Statistical Significance: While the multi-turn approach showed improvements, some comparisons (e.g., full list vs. subset of MeSH) were not statistically significant.
  • Future Directions: The authors plan to:
  • Integrate automatic feedback into the learning process to further enhance effectiveness.
  • Conduct human evaluations to better assess summary alignment with human judgments.
  • Explore both closed- and open-source LLMs to assess generalizability across different models.
  • Further refine MeSH prediction models and term selection strategies.

Summary updated

BibTeX
@inproceedings{ming2025towards,
  author = {Shufan Ming and Yue Guo and Halil Kilicoglu},
  title = {{Towards Knowledge-Guided Biomedical Lay Summarization using Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure