Introduction
Radiology reports are essential for clinical decision-making but are often inaccessible to patients due to dense medical jargon and a lack of visual context. As patient portals provide direct access to these records, the gap between data availability and patient understanding has widened. With a global shortage of radiologists, personalized walk-throughs of imaging findings are increasingly rare. ReXplain addresses this by automating the creation of patient-friendly video reports, combining large language models (LLMs), medical image segmentation, and digital avatars to simulate a professional consultation.
Methodology
The ReXplain pipeline integrates several advanced AI modules to transform raw 3D CT data and unstructured text into a cohesive video presentation.
- Text Simplification and Mapping: Utilizing GPT-4o, the system extracts positive findings from radiology reports and translates them into lay language. It simultaneously maps these findings to a library of 201 anatomical regions.
- Anatomical Grounding: The "Segment Anything in medical images via Text" (SAT) model is employed to automatically segment and highlight the relevant organs within the 3D CT volume based on the LLM's mapping.
- Comparative Visualization: To aid understanding, the system retrieves a "normal" reference scan from the CT-RATE dataset and registers it to the patient's scan, allowing for side-by-side comparison.
- Avatar Generation: Using the Tavus platform and 3D Gaussian Splatting, a realistic virtual radiologist is generated to narrate the report.
- Automated Assembly: The final output is a structured video that introduces findings with bounding boxes, scrolls through CT slices, and provides comparative visual evidence.
Results and Key Insights
The system was evaluated using the CT-RATE dataset, a large-scale collection of chest CT volumes and reports. Technical evaluation showed that GPT-4o achieved 100% precision in mapping findings to anatomical regions, with an 85% recall rate for identifying all abnormalities.
A pilot study involving six board-certified radiologists validated the system's utility. Over 64% of participants rated the accuracy and clarity of the generated information positively. Notably, the "layperson explanation" and "comparison with normal images" features received over 74% positive feedback. Radiologists highlighted that showing a "normal" benchmark is one of the most effective ways to communicate abnormalities to non-experts.
Beyond technical accuracy, the researchers emphasized the importance of delivery. The use of a digital avatar aims to simulate the experience of a one-on-one consultation, recognizing that the human-like delivery is as important for patient engagement as the data itself.
Discussion and Future Work
While ReXplain demonstrates the feasibility of automated video reporting, several areas for improvement remain. Current segmentation is organ-level; future iterations require "lesion-level" granularity to pinpoint small nodules. Additionally, improving video navigation—such as strategic pausing or interactive slice selection—would enhance the patient experience.
ReXplain is envisioned as a "pre-consultation" tool to prepare patients for discussions with their physicians, rather than a replacement for human interaction. Future research will focus on expanding the system to other modalities, such as MRI and ultrasound, to broaden its clinical impact.
Summary updated