ReXplain: Translating Radiology into Patient-Friendly Video Reports

Luyang Luo, Jenanan Vairavamurthy, Xiaoman Zhang, Abhinav Kumar, Ramon R. Ter-Oganesyan, Stuart T. Schroff, Dan Shilo, Rydhwana Hossain, Mike Moritz, Pranav RajpurkararXiv 2024 · DOI · arXiv

Introduction

Radiology reports are essential for clinical decision-making but are often inaccessible to patients due to dense medical jargon and a lack of visual context. As patient portals provide direct access to these records, the gap between data availability and patient understanding has widened. With a global shortage of radiologists, personalized walk-throughs of imaging findings are increasingly rare. ReXplain addresses this by automating the creation of patient-friendly video reports, combining large language models (LLMs), medical image segmentation, and digital avatars to simulate a professional consultation.

Methodology

The ReXplain pipeline integrates several advanced AI modules to transform raw 3D CT data and unstructured text into a cohesive video presentation.

  1. Text Simplification and Mapping: Utilizing GPT-4o, the system extracts positive findings from radiology reports and translates them into lay language. It simultaneously maps these findings to a library of 201 anatomical regions.
  2. Anatomical Grounding: The "Segment Anything in medical images via Text" (SAT) model is employed to automatically segment and highlight the relevant organs within the 3D CT volume based on the LLM's mapping.
  3. Comparative Visualization: To aid understanding, the system retrieves a "normal" reference scan from the CT-RATE dataset and registers it to the patient's scan, allowing for side-by-side comparison.
  4. Avatar Generation: Using the Tavus platform and 3D Gaussian Splatting, a realistic virtual radiologist is generated to narrate the report.
  5. Automated Assembly: The final output is a structured video that introduces findings with bounding boxes, scrolls through CT slices, and provides comparative visual evidence.
The ReXplain pipeline components: (a) CT finding localization, (b) reference comparison, (c) 3D organ segmentation, and (d) digital avatar narration.

Results and Key Insights

The system was evaluated using the CT-RATE dataset, a large-scale collection of chest CT volumes and reports. Technical evaluation showed that GPT-4o achieved 100% precision in mapping findings to anatomical regions, with an 85% recall rate for identifying all abnormalities.

A pilot study involving six board-certified radiologists validated the system's utility. Over 64% of participants rated the accuracy and clarity of the generated information positively. Notably, the "layperson explanation" and "comparison with normal images" features received over 74% positive feedback. Radiologists highlighted that showing a "normal" benchmark is one of the most effective ways to communicate abnormalities to non-experts.

A digital avatar generated via 3D Gaussian Splatting to simulate a face-to-face radiologist consultation.

Beyond technical accuracy, the researchers emphasized the importance of delivery. The use of a digital avatar aims to simulate the experience of a one-on-one consultation, recognizing that the human-like delivery is as important for patient engagement as the data itself.

Discussion and Future Work

While ReXplain demonstrates the feasibility of automated video reporting, several areas for improvement remain. Current segmentation is organ-level; future iterations require "lesion-level" granularity to pinpoint small nodules. Additionally, improving video navigation—such as strategic pausing or interactive slice selection—would enhance the patient experience.

ReXplain is envisioned as a "pre-consultation" tool to prepare patients for discussions with their physicians, rather than a replacement for human interaction. Future research will focus on expanding the system to other modalities, such as MRI and ultrasound, to broaden its clinical impact.

Summary updated

Abstract

Radiology reports, designed for efficient communication between medical experts, often remain incomprehensible to patients. This inaccessibility could potentially lead to anxiety, decreased engagement in treatment decisions, and poorer health outcomes, undermining patient-centered care. We present ReXplain (Radiology eXplanation), an innovative AI-driven system that translates radiology findings into patient-friendly video reports. ReXplain uniquely integrates a large language model for medical text simplification and text-anatomy association, an image segmentation model for anatomical region identification, and an avatar generation tool for engaging interface visualization. ReXplain enables producing comprehensive explanations with plain language, highlighted imagery, and 3D organ renderings in the form of video reports. To evaluate the utility of ReXplain-generated explanations, we conducted two rounds of user feedback collection from six board-certified radiologists. The results of this proof-of-concept study indicate that ReXplain could accurately deliver radiological information and effectively simulate one-on-one consultation, shedding light on enhancing patient-centered radiology with potential clinical usage. This work demonstrates a new paradigm in AI-assisted medical communication, potentially improving patient engagement and satisfaction in radiology care, and opens new avenues for research in multimodal medical communication.
BibTeX
@misc{luo2024rexplain,
  author = {Luyang Luo and Jenanan Vairavamurthy and Xiaoman Zhang and Abhinav Kumar and Ramon R. Ter-Oganesyan and Stuart T. Schroff and Dan Shilo and Rydhwana Hossain and Mike Moritz and Pranav Rajpurkar},
  title = {{ReXplain: Translating Radiology into Patient-Friendly Video Reports}},
  year = {2024},
  eprint = {2410.00441},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.00441},
}
Figure