Introduction
While computer vision has seen transformative success with multimodal encoding and interactive large language models (LLMs), medical imaging AI—particularly in the realm of 3D imaging—has lagged behind. The primary bottleneck is a "data desert": a scarcity of large-scale, open-source datasets that pair 3D volumes with descriptive textual reports. Most existing models are either task-specific or rely on 2D adaptations that fail to capture the complex spatial context of 3D Computed Tomography (CT) scans.
To address this gap, researchers have introduced CT-RATE, the first large-scale public dataset pairing 3D chest CT volumes with radiology reports. Leveraging this data, they developed CT-CLIP and CT-CHAT, foundation models that achieve supervised-level performance in zero-shot abnormality detection and provide a conversational interface for 3D medical imaging.
The CT-RATE Dataset
At the core of this advancement is CT-RATE, a comprehensive dataset comprising 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Through various reconstructions, these scans are expanded to 50,188 volumes, totaling over 14.3 million 2D slices. Each scan is paired with its corresponding radiology report, providing a rich source of natural language supervision that allows models to learn "medical common sense" rather than relying on rigid, manually annotated binary labels.
Methodology: CT-CLIP and CT-CHAT
CT-CLIP (Contrastive Language-Image Pretraining)
Inspired by the CLIP framework, CT-CLIP aligns 3D image features with textual descriptions. It utilizes a 3D Vision Transformer (incorporating spatial and causal transformers to handle the depth of CT scans) and a text transformer. This architecture allows the model to understand 3D volumes as a whole, preserving critical anatomical relationships that slice-by-slice 2D models often miss.
To further refine the model, the researchers introduced VocabFine, a novel open-vocabulary finetuning method. VocabFine improves accuracy on specific tasks while ensuring the model retains its ability to recognize and describe abnormalities it was not explicitly trained on during the finetuning phase.
CT-CHAT
By connecting the CT-CLIP vision encoder to a Large Language Model (Llama 3.1) via a multimodal projector, the team created CT-CHAT. This conversational assistant was finetuned using LoRA (Low-Rank Adaptation) on over 2.7 million medical question-answer pairs derived from the CT-RATE dataset (CT-VQA). CT-CHAT allows clinicians to interact with 3D volumes through a natural language interface, querying specific findings or requesting report summaries.
Key Results and Insights
Supervised-Level Zero-Shot Performance
One of the most significant findings is that the zero-shot CT-CLIP model—which never saw explicit labels during training—outperformed state-of-the-art fully supervised models (such as CT-Net) across all key metrics. On internal validation, CT-CLIP achieved a mean AUROC of 0.731, compared to the supervised baseline of 0.629. This suggests that natural language provides a more powerful and nuanced signal for learning medical features than traditional classification labels.
Generalization and Clinical Accuracy
To ensure the model's robustness, it was tested on external datasets (RAD-ChestCT and UPMC) from different institutions and countries. CT-CLIP maintained its lead, showing an AUROC gain of 0.085 over supervised models.
In terms of report generation, a blinded study involving three board-certified radiologists found that CT-CHAT’s reports were consistently superior to existing 3D medical models like RadFM and CT2Rep in clinical accuracy and detail. Furthermore, CT-CHAT demonstrated a significant leap in clinical accuracy metrics (Llama Scores) compared to 2D-based assistants like LLaVA-Med.
Limitations and Future Directions
While these models represent a breakthrough, several challenges remain:
- Single-Center Bias: The training data originates from one major hospital system, which may reflect specific local scanning protocols.
- Hallucination Risks: As with all LLMs, CT-CHAT can occasionally hallucinate findings. While system prompts and guardrails have been implemented, clinical deployment requires even higher reliability standards.
- Anatomical Scope: The current models are specialized for chest CTs. Future work aims to expand the CT-RATE dataset to include other body regions (abdomen, head) and additional modalities such as MRI and PET scans.
Conclusion
The open-source release of CT-RATE, CT-CLIP, and CT-CHAT addresses critical challenges in 3D medical imaging. By bridging the gap between 2D and 3D AI, these foundation models lay the groundwork for more intuitive, accurate, and generalizable tools in medical diagnostics and patient care.
Reference: Hamamci, I. E., et al. (2024). Generalist foundation models from a multimodal dataset for 3D computed tomography. arXiv:2403.17834v5.
Summary updated