Retrieval Augmented Therapy Suggestion for Molecular Tumor Boards: Algorithmic Development and Validation Study

Eliza Berman, Holly Sundberg Malek, Michael Bitzer, Nisar P. Malek, Carsten EickhoffJournal of Medical Internet Research 2025 · DOI

TL;DR

This paper explores using Retrieval Augmented Generation (RAG) with Large Language Models (LLMs) to assist Molecular Tumor Boards (MTBs) in generating cancer treatment recommendations. While RAG-enhanced LLMs show promise in accelerating the process and sometimes producing clinician-level suggestions, they still require significant human oversight due to issues like hallucinations and inconsistent citation accuracy.

What Problem Does This Paper Address?

Molecular Tumor Boards (MTBs) are crucial for precision oncology, bringing together experts to interpret a patient's genomic profile and recommend optimal treatments. However, this process is incredibly time-consuming and labor-intensive, involving manual review of vast amounts of data from various databases. This bottleneck limits the number of patients that can be effectively reviewed, potentially delaying care. The paper aims to address this by investigating how LLMs, specifically through a RAG approach, can automate and accelerate the generation of evidence-based treatment recommendations for MTB patients.

Methods

The core of the methodology involves building a Retrieval Augmented Generation (RAG) pipeline. This pipeline integrates a large language model (LLM), specifically Llama-2-7b-chat, with a retrieval system that queries PubMed for relevant medical literature.

Here's a breakdown of the process:

  • Data Parsing and Query Construction: Patient diagnoses and relevant genetic mutations are extracted from MTB case materials. These are then used to construct two types of queries: one for document retrieval (broader cancer type + gene mutation) and another for LLM inference (more detailed patient information + gene mutation, asking for the best therapy and PubMed citations).
  • Document Retrieval: The PubMed Data Loader, orchestrated by LlamaIndex, queries PubMed using the constructed retrieval queries. LlamaIndex embeds the retrieved documents into a vector store, enabling efficient searching.
  • LLM Inference: The retrieved documents are fed into the LLM along with the inference query. The LLM then generates a treatment recommendation, justification, and references.
  • Evaluation: The generated recommendations are compared against historical MTB protocols (ground truth). Additionally, clinicians manually assess the relevance and correctness of the LLM responses and their cited articles.

Datasets

The study utilizes historical case reports from Molecular Tumor Board (MTB) conferences at the Center for Personalized Medicine at the Universitätsklinikum Tübingen for evaluation. The retrieval component of the RAG system relies on PubMed as its primary data source. No new benchmark datasets were introduced; instead, existing historical data was adapted for evaluation.

Key Insights and Ideas

  • RAG as a Promising Accelerator: The RAG approach significantly streamlines the evidence retrieval and recommendation generation process, offering a potential solution to the time constraints of traditional MTBs.
  • Clinician Input is Crucial: Prompting the LLM with clinician-crafted queries yielded more accurate responses and relevant citations compared to automatically generated queries, highlighting the continued importance of human expertise in guiding the AI.
  • LLM Memory vs. Retrieval: The study observed that LLM responses sometimes didn't perfectly align with retrieved references, suggesting that the LLM's internal knowledge ("memory") plays a role alongside the retrieved information. This implies a need to better understand and control this interplay.
  • Hallucinations Remain a Challenge: Despite RAG, a significant percentage of cited articles were either hallucinations or not found on PubMed, underscoring the persistent issue of LLM unreliability in critical applications.
  • Supervision is Non-Negotiable: The findings strongly indicate that LLMs, in their current state, cannot fully automate MTB processes. Heavy clinician supervision is essential at multiple stages, from query generation to final recommendation review.

Key Results

  • Citation Accuracy: 75% of referenced articles were correctly cited from PubMed. However, 17% were hallucinations, and the remaining were not properly cited.
  • Clinician-Equal Recommendations: Clinician-generated LLM queries resulted in recommendations that were considered equal to clinician recommendations in 25% of cases and alternative plausible treatments in 38% of cases.
  • Article Relevance: 87% of retrieved articles were deemed "not relevant" when using automated queries, dropping to 67.78% with clinician-generated queries.
  • LLM Response Feedback: Clinician feedback categorized 37.5% of LLM responses as "Off-base" (nonsensical), 37.5% as "Alternative" (plausible but not the primary suggestion), and 25% as "Reference" (matching the MTB conference suggestion) when using clinician-generated queries.

Limitations and Future Work

Limitations:

  • Reproducibility: LLM responses can be dynamic and inconsistent, even with identical prompts, making exact replication challenging.
  • Scope of Queries: The current system only queries genetic mutations with evidence levels 1-3 on OncoKB, excluding other potentially relevant biomarkers and patient characteristics (e.g., tumor mutation burden, HLA expression).
  • Evaluation Technique: The clinician feedback suggested a need for an additional category to account for LLM responses that suggest combination treatments of multiple drugs.
  • Generalizability: The study was conducted at a single cancer center, limiting its generalizability to other institutions and healthcare settings.

Future Work:

  • Local Hospital Data Integration: Incorporating local hospital patient history and MTB conference recommendations into the RAG system could improve accuracy and context-specificity.
  • Isolated IR and LLM Experimentation: Separating the information retrieval (IR) step from LLM generation could ensure higher relevance of retrieved documents before they are fed to the LLM.
  • Prompting Strategies: Further research into diverse prompting strategies (e.g., ranking treatments, asking for multiple options, chain-of-thought prompting) could enhance LLM performance.
  • Interpretability and Transparency: Developing methods to understand why an LLM makes a certain recommendation (e.g., using Shapley values) is crucial for building clinician trust.
  • Efficiency Analysis: Quantifying the actual time and resource savings achieved by the RAG-enhanced LLM pipeline compared to traditional MTB workflows is a key next step.
  • Mitigating Hallucinations: Developing robust strategies to reduce and mitigate LLM hallucinations is imperative for safe clinical deployment.

Summary updated

Abstract

This study demonstrates how retrieval augmented generation-enhanced LLMs can be a powerful tool in accelerating MTB conferences, as LLMs are sometimes capable of achieving clinician-equal treatment recommendations. However, further investigation is required to achieve stable results with zero hallucinations. LLMs signify a scalable solution to the time-intensive process of MTB investigations. However, LLM performance demonstrates that they must be used with heavy clinician supervision, and cannot yet fully automate the MTB pipeline.
BibTeX
@article{berman2025retrieval,
  author = {Eliza Berman and Holly Sundberg Malek and Michael Bitzer and Nisar P. Malek and Carsten Eickhoff},
  title = {{Retrieval Augmented Therapy Suggestion for Molecular Tumor Boards: Algorithmic Development and Validation Study}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/64364},
}
Figure