Expertly Informed, Generatively Summarized: A Hybrid RAG Approach to Informed Consent Summarization with Auxiliary Expert Knowledge

Autumn Toney-Wails, Ryan Wails, Caleb SmithNAACL 2025

TL;DR

This paper introduces a novel hybrid Retrieval Augmented Generation (RAG) and GraphRAG system designed to automatically summarize the "Key Information" section of Institutional Review Board (IRB) informed consent documents. By integrating expert knowledge and a knowledge graph, the system aims to produce accurate, clear, and risk-differentiated summaries, addressing a critical need for reliable AI in high-stakes medical communication.

What Problem Does This Paper Address?

Informed consent documents (ICDs) are crucial for potential participants in medical studies, outlining study details, risks, and benefits. However, these documents are often lengthy and technical, making it challenging for individuals to fully grasp the implications of participation. While Generative AI (GenAI) shows promise in summarizing complex information, ensuring accuracy and avoiding misrepresentation in high-risk medical contexts is paramount. Specifically, accurately differentiating between risks inherent to the standard of care versus those introduced by the research itself is a significant challenge. This paper tackles the problem of generating reliable, expert-informed, and easily understandable summaries of the "Key Information" section of ICDs, aiming to improve participant comprehension and decision-making.

Methods

The authors developed a hybrid RAG and GraphRAG system. The core RAG component takes a user-uploaded informed consent document as its knowledge base, processes it in real-time (extracting, chunking, and vectorizing text), and generates summaries. To address the critical challenge of differentiating standard of care risks from research risks, they introduced a Temporary Auxiliary GraphRAG (TAGRAG) component. TAGRAG leverages a knowledge graph built from relevant medical research publications (retrieved via PubMed) to provide expert context. This allows the system to query for information on standard of care and compare it with risks identified in the RAG component of the ICD. The system underwent 11 iterations of design and refinement, incorporating feedback from subject matter experts (SMEs) and principal investigators (PIs).

Datasets

The study utilized 18 human-authored informed consent documents (ICDs) from an institution's medical school IRB, covering various study types (clinical trials for drugs and medical devices, data registries, cancer, and pediatric populations). For the TAGRAG component, the system queried the PubMed Central (PMC) dataset, specifically focusing on research publications related to the standard of care for a given disease.

Key Insights and Ideas

  • Hybrid RAG + GraphRAG for High-Stakes Summarization: Combining the direct summarization capabilities of RAG with the structured, expert-informed context of GraphRAG is crucial for generating reliable summaries in domains like medical consent.
  • The Importance of Risk Differentiation: Accurately distinguishing between risks associated with standard medical treatment and those unique to a research study is a complex but vital task for informed consent. The TAGRAG component was specifically designed to tackle this.
  • Iterative Refinement with Expert Feedback: The system's development involved extensive iteration based on feedback from IRB SMEs and clinical trial PIs, highlighting the necessity of human oversight and domain expertise in AI development for sensitive applications.
  • Balancing Clarity and Accuracy: The challenge lies in presenting complex medical information in a clear, lay-friendly manner without sacrificing factual accuracy or essential details.

Key Results

  • SME Evaluations: The final model achieved an accuracy of 0.72 for "Style and Structure" and 0.7 for "Factual Accuracy" according to SME ratings. "Information Weighting" and "Standard of Care vs. Research Differentiation" were the lowest performing metrics at 0.63.
  • PI Evaluations: 78% of PIs found the AI-generated summaries factually accurate regarding the research nature. 100% found the anticipated benefits accurately described, and 75% found the potential risks accurately described. 87.5% of PIs indicated they would likely use the tool to draft KI sections in the future.
  • Accuracy Improvement: The system demonstrated improved factual accuracy over iterations, increasing from 0.3 in version 1 to 0.7 in the final model.

Limitations and Future Work

  • Risk Differentiation Performance: While improved, risk differentiation remained a challenging area, with SMEs rating it at 0.63. The authors acknowledge this as a key area for future work.
  • Generalizability: While the system is designed for generalizability, its current implementation is specific to IRB informed consent documents. Adapting the hybrid RAG and TAGRAG pipeline to other domains is a potential future direction.
  • AI Limitations: The authors emphasize that the AI-generated summaries are drafts requiring review, and users must ensure accuracy and appropriateness. They also note the ethical considerations of using closed-source LLMs for medical tasks.
  • Future Work: The paper suggests further research into improving risk differentiation and exploring the adaptation of the hybrid system to other domains.

Summary updated

BibTeX
@inproceedings{toneywails2025expertly,
  author = {Autumn Toney-Wails and Ryan Wails and Caleb Smith},
  title = {{Expertly Informed, Generatively Summarized: A Hybrid RAG Approach to Informed Consent Summarization with Auxiliary Expert Knowledge}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure