Summarizing clinical evidence utilizing large language models for cancer treatments: a blinded comparative analysis

Samuel M. Rubinstein, Aleenah Mohsin, Rahul Banerjee, Will Ma, Sanjay Mishra, Mary Kwok, Peter Yang, Jeremy L. Warner, Andrew J. CowanFrontiers Digit. Health 2025 · DOI

TL;DR

This study evaluates four leading Large Language Models (LLMs) for their ability to summarize complex clinical evidence for cancer treatments. While Claude 3.5 showed the most promise, all LLMs required significant expert editing, highlighting the current limitations of AI in generating reliable clinical synopses independently.

What Problem Does This Paper Address?

Clinicians and guideline developers need concise summaries of the ever-growing body of medical literature to make informed treatment decisions. Manually curating this information is time-consuming and labor-intensive. Large Language Models (LLMs) offer a potential solution by automating summarization, but concerns remain about their accuracy, completeness, and tendency to "hallucinate" (generate fabricated information). This paper addresses the critical need to objectively assess the capabilities of commercially available LLMs for this task, specifically in the context of cancer treatment evidence, to understand if their benefits outweigh the risks.

Methods

The researchers selected four prominent LLMs: Claude 3.5, ChatGPT 4.0, Gemini 1.0, and Llama 2. They developed prompts to generate synopses for six specific cancer treatment regimens in multiple myeloma and AL amyloidosis. These synopses were then independently reviewed by two board-certified hematologists using a 5-point Likert scale for accuracy, completeness, relevance, clarity, and coherence. Hallucinations were assessed on a 4-point ordinal scale. Inter-rater reliability was measured using Cohen's quadratic weighted kappa. The study also assessed the proportion of synopses requiring minimal editing.

Datasets

The study did not use datasets for training LLMs. Instead, it evaluated the LLMs' ability to summarize existing clinical evidence. The "dataset" for evaluation consisted of synopses generated by the LLMs for six specific treatment regimens in multiple myeloma and AL amyloidosis. These regimens were:

  1. Daratumumab, lenalidomide, bortezomib, and dexamethasone (Dara-VRd)
  2. Carfilzomib, lenalidomide, and dexamethasone (KRd)
  3. Bortezomib, thalidomide, dexamethasone, cisplatin, doxorubicin, cyclophosphamide, and etoposide (VTD-PACE)
  4. Daratumumab, cyclophosphamide, bortezomib, and dexamethasone (Dara-CyBorD)
  5. Elranatamab monotherapy
  6. Talquetamab monotherapy

Key Insights and Ideas

  • LLMs are not yet ready for independent clinical summarization: Despite advancements, all tested LLMs required substantial editing by domain experts to be considered usable. This underscores the continued necessity of human oversight in critical medical information synthesis.
  • Claude 3.5 leads the pack, but still falls short: Claude 3.5 consistently outperformed the other LLMs across most evaluated domains, particularly in accuracy and completeness. However, even its performance wasn't perfect, and it still needed editing.
  • Hallucinations remain a significant concern: While Claude 3.5 showed minimal hallucinations, other LLMs, particularly Llama 2, exhibited a notable tendency to generate inaccurate or fabricated information, which is a critical safety issue in medicine.
  • Inter-rater reliability varies: Agreement between expert reviewers was strongest for well-established treatment regimens, suggesting LLMs might be more reliable for summarizing older, more standardized evidence. Agreement was weaker for newer agents and subjective criteria like clarity and coherence.
  • Nuance and context are missing: LLMs struggled to capture the subtle but crucial clinical judgment needed to account for patient-specific variables or to accurately represent the limitations and specific patient populations of clinical trials.

Key Results

  • Overall Performance: Claude 3.5 achieved the highest mean Likert scores across all domains, with accuracy at 3.92 and completeness at 4.00. Llama 2 consistently performed the poorest, with accuracy at 1.92. ChatGPT 4.0 and Gemini 1.0 showed intermediate performance.
  • Minimal Editing Required: Only 66.7% of Claude's synopses required minimal editing, compared to 33.3% for GPT4, 16.7% for Gemini, and 8.3% for Llama.
  • Hallucinations: Claude 3.5 had a mean Likert score of 4.00 for hallucinations (indicating few to no hallucinations), while Llama 2 scored 1.92.
  • Inter-rater reliability: Moderate to substantial agreement was observed for accuracy (κ=0.649) and relevance (κ=0.761), but fair to poor agreement for completeness (κ=0.521), hallucinations (κ=0.362), coherence (κ=0.353), and clarity (κ=0.135).

Limitations and Future Work

The authors acknowledge several limitations:

  • Usability without editing: Many synopses were not usable without significant editing, which limits their immediate practical application.
  • Time comparison: The study did not compare the time saved by using LLMs versus manual summarization.
  • LLM iterations: They could not assess different versions or iterations of the same LLM (e.g., GPT3 vs. GPT4).

Future work suggested includes:

  • Specialized LLMs: Developing domain-adapted LLMs trained on curated clinical corpora and real-world patient data.
  • Retrieval-Augmented Generation (RAG): Exploring RAG and fine-tuning techniques to improve LLM performance, though these require specific expertise and resources.
  • Ensemble/Agentic Approaches: Investigating combinations of LLMs or more sophisticated AI architectures to overcome individual model limitations.
  • Integration into Workflows: Exploring the integration of LLMs into healthcare IT infrastructure, such as Electronic Health Records (EHRs), to streamline clinical workflows.

Summary updated

BibTeX
@article{rubinstein2025summarizing,
  author = {Samuel M. Rubinstein and Aleenah Mohsin and Rahul Banerjee and Will Ma and Sanjay Mishra and Mary Kwok and Peter Yang and Jeremy L. Warner and Andrew J. Cowan},
  title = {{Summarizing clinical evidence utilizing large language models for cancer treatments: a blinded comparative analysis}},
  journal = {Frontiers Digit. Health},
  year = {2025},
  doi = {10.3389/fdgth.2025.1569554},
}
Figure