Towards Understanding LLM-Generated Biomedical Lay Summaries

Rohan Charudatt Salvi, Shrikant Panigrahi, Dhruv Jain, Shweta Yadav, Md Shad Akhtar2025 · DOI

Abstract

In this paper, we investigate the effectiveness of large language models in generating accessible lay summaries of medical abstracts, targeting non-expert audiences. We assess the ability of models like GPT-4, Biomistral, and LLaMA 3-8B-Instruct to simplify complex medical information, focusing on layness, comprehensiveness, and factual accuracy. Utilizing both automated and human evaluations, we discover that automatic metrics do not always align with human judgments. Our analysis highlights the potential benefits of developing clear guidelines for consistent evaluations conducted by non-expert reviewers. It also points to areas for improvement in the evaluation process and the creation of lay summaries for future research.

Introduction

Medical research is often characterized by specialized terminology and complex sentence structures, creating a barrier for patients, caregivers, and journalists. Lay summaries—simplified versions of research abstracts—aim to bridge this gap. However, manual creation is resource-intensive. While Large Language Models (LLMs) show promise in summarization, the biomedical domain requires a delicate balance of "layness" (simplicity), "comprehensiveness" (completeness), and "factuality" (accuracy). This study investigates whether current LLMs can achieve this balance and if standard automated metrics are sufficient for evaluation.

Methodology

The researchers evaluated four prominent LLMs: GPT-4, Mistral-large, LLaMA 3-8B-Instruct, and BioMistral (a domain-specific model). A few-shot prompting strategy was employed, instructing models to explain research to a 10th-grade audience.

Evaluation was conducted through two lenses:

  1. Automated Metrics: Including ROUGE (text overlap), SARI (simplification), and readability scores like FKGL and LENS.
  2. Human Evaluation: Two non-expert evaluators rated summaries on a 1–5 Likert scale for Comprehensiveness, Layness, and Factuality, guided by a new set of standardized manual evaluation instructions.

The study utilized the PLABA dataset (plain language adaptation) and the PLOS dataset (author-written lay summaries).

Results and Discussion

Automated vs. Human Evaluation

A significant finding is the "Metric Misalignment" between automated scores and human judgment. Fine-tuned models like BioMistral achieved the highest ROUGE scores by staying close to the source text. However, human evaluators preferred models like GPT-4 and Mistral, which were more "creative" in their simplification.

Table 1: Automated evaluation metrics across different models and methods.

As shown in Table 1, the fine-tuned BioMistral model dominated ROUGE-1 scores (0.634), whereas Mistral-large scored lower (0.528). Conversely, GPT-4 excelled in the LENS metric (74.958), which specifically targets simplification quality.

Human Preference and the Simplicity Trade-off

Human evaluation revealed that Mistral-large and GPT-4 were superior in generating accessible content. On the PLABA dataset, Mistral achieved a Layness score of 4.77/5, compared to only 3.105 for the BioMistral baseline.

Table 2: Human evaluation scores for Comprehensiveness, Layness, and Factuality across PLABA and PLOS datasets.

Table 2 highlights that while Mistral led in layness and comprehensiveness, there is an inherent tension with factuality. To achieve high "layness," models sometimes omit nuance or introduce "extrinsic hallucinations"—adding context not present in the source. While often helpful for clarity, these additions pose risks in a medical context.

Conclusion

The study underscores that traditional NLP metrics like ROUGE often penalize the very "creativity" (e.g., adding definitions or analogies) that makes a summary useful for non-experts. While LLMs like Mistral and GPT-4 show great potential, the risk of hallucinations necessitates further research into evaluation frameworks that combine human perception with expert medical verification. Future work will focus on developing metrics that better align with human satisfaction and safety.

Summary updated

Abstract

In this paper, we investigate the effectiveness of large language models in generating accessible lay summaries of medical abstracts, targeting non-expert audiences.We assess the ability of models like GPT-4, Biomistral, and LLaMA 3-8B-Instruct to simplify complex medical information, focusing on layness, comprehensiveness, and factual accuracy.Utilizing both automated and human evaluations, we discover that automatic metrics do not always align with human judgments.Our analysis highlights the potential benefits of developing clear guidelines for consistent evaluations conducted by non-expert reviewers.It also points to areas for improvement in the evaluation process and the creation of lay summaries for future research.
BibTeX
@misc{salvi2025towards,
  author = {Rohan Charudatt Salvi and Shrikant Panigrahi and Dhruv Jain and Shweta Yadav and Md Shad Akhtar},
  title = {{Towards Understanding LLM-Generated Biomedical Lay Summaries}},
  year = {2025},
  doi = {10.18653/v1/2025.cl4health-1.22},
}
Figure