Specialty-Oriented Generalist Medical AI for Chest CT Screening

Chuang Niu, Qing Lyu, Christopher D. Carothers, Parisa Kaviani, Josh Tan, Pingkun Yan, Mannudeep K. Kalra, Christopher T. Whitlow, Ge WangarXiv 2023 · arXiv

TL;DR

Researchers have developed M3FM, the first "Specialty-Oriented Generalist Medical AI" (SOGMAI) designed specifically for lung cancer screening. By integrating 3D CT scans with clinical text through a unified question-answering framework, M3FM outperforms specialized state-of-the-art models across 17 different medical tasks, including lung cancer risk prediction and cardiovascular disease mortality estimation.

What Problem Does This Paper Address?

Current medical AI is highly fragmented. Most models are "specialists"—designed for a single task (like detecting a lung nodule) using a single data type (like a CT scan). However, real-world clinical practice is inherently multimodal and multitask. A radiologist doesn't just look at an image; they interpret it in the context of a patient’s smoking history, laboratory results, and previous diagnoses.

The authors identify two major hurdles to creating a "Generalist Medical AI" (GMAI):

  1. The Data Challenge: Curating massive, aligned datasets that link high-dimensional 3D tomographic scans with unstructured clinical text across thousands of patients.
  2. The Model Challenge: Designing a scalable architecture that can process these diverse inputs and handle multiple clinical questions simultaneously without losing critical information (like the actual physical size of a tumor).

Methods

The authors propose the Medical Multimodal-Multitask Foundation Model (M3FM). Its architecture consists of four main pillars:

  • CTViT (CT Vision Transformer): A novel image encoder designed for 3D volumes. Unlike standard vision transformers that might lose scale information during resizing, CTViT uses a "physical size embedding." It treats the actual voxel dimensions (in millimeters) as a separate input, ensuring the model knows exactly how large a finding is in the real world.
  • Text Transformer: A RoBERTa-based encoder that processes clinical history and the specific "question" or task prompt.
  • Task Encoder & Predictors: A unified framework that fuses image and text features. By formulating tasks as Multimodal Question-Answering (MQA), the model can be "prompted" to perform different tasks—such as "Is there emphysema?" or "Predict the 6-year lung cancer risk"—using the same underlying weights.

The model was trained using a two-stage process: large-scale self-supervised pre-training (masked image modeling) followed by multitask supervised fine-tuning.

M3FM inference illustration showing how 3D CT and clinical text are processed through a unified encoder-predictor pipeline.

Datasets

The study introduces OpenM3Chest, which the authors claim is the largest publicly available 3D CT dataset for MQA.

  • Scale: 163,725 chest CT series from 26,254 patients.
  • Sources: Curated from the National Lung Screening Trial (NLST), MIDRC, and independent cohorts from Wake Forest University (WFUSM) and Massachusetts General Hospital (MGH).
  • Diversity: Includes 49 clinical data types (demographics, smoking history, pathology, etc.) aligned with 17 distinct tasks ranging from nodule characterization to COVID-19 detection.

Key Insights and Ideas

  • SOGMAI vs. GMAI: The authors argue that the path to a truly general medical AI is through "Specialty-Oriented" models. By focusing on a specific clinical workflow (Lung Cancer Screening), they can build a model that is general enough to handle all related tasks while remaining specific enough to be clinically useful.
  • Physical Size is a First-Class Citizen: In medical imaging, size is diagnostic. The authors found that explicitly embedding the physical dimensions of voxels significantly boosted performance in tasks like nodule characterization and risk prediction.
  • Multimodal Synergy: The model doesn't just use text as a label; it uses it as context. For example, CVD mortality prediction improved by ~10% when the model had access to both CT scans and clinical history compared to scans alone.
  • Unified Prompting: The MQA framework allows the model to mimic a radiologist’s workflow—receiving a "task command" and paying attention to relevant multimodal data to provide an answer.

Key Results

M3FM set new records across the board, consistently outperforming specialized models like Sybil (for lung cancer risk) and Tri2D-Net (for CVD risk).

  • Lung Cancer Risk: Improved prediction accuracy (AUC) by up to 20% on independent datasets compared to previous SOTA models.
  • CVD Mortality: Achieved up to a 10% improvement in risk estimation.
  • Generalizability: The model showed strong performance on "out-of-distribution" tasks, such as predicting immunotherapy-induced pneumonitis, through simple transfer learning.
  • Scaling: As shown in Figure 3, increasing the model size from "Base" (257M parameters) to "Huge" (865M parameters) consistently improved performance, following the "scaling laws" seen in LLMs.
Performance comparison showing M3FM (blue) outperforming previous SOTA models across various tasks.

Limitations and Future Work

While impressive, the authors note several limitations:

  • Retrospective Nature: The study was conducted on historical data; prospective, real-world clinical trials are needed to assess actual impact on patient outcomes.
  • Integration Hurdles: Deploying such a model requires real-time integration across disparate hospital systems (Electronic Medical Records, PACS, etc.), which remains a significant logistical challenge.
  • Data Quality: While the model scales well, the authors noted diminishing returns when moving from "Large" to "Huge" models, suggesting that even larger and higher-quality datasets are required to unlock further gains.

Future work will focus on expanding this SOGMAI approach to other medical specialties and integrating more complex longitudinal data, such as surgical and radiation treatment histories.

Summary updated

Abstract

Modern medical records include a vast amount of multimodal free text clinical data and imaging data from radiology, cardiology, and digital pathology. Fully mining such big data requires multitasking; otherwise, occult but important aspects may be overlooked, adversely affecting clinical management and population healthcare. Despite remarkable successes of AI in individual tasks with single-modal data, the progress in developing generalist medical AI remains relatively slow to combine multimodal data for multitasks because of the dual challenges of data curation and model architecture. The data challenge involves querying and curating multimodal structured and unstructured text, alphanumeric, and especially 3D tomographic scans on an individual patient level for real-time decisions and on a scale to estimate population health statistics. The model challenge demands a scalable and adaptable network architecture to integrate multimodal datasets for diverse clinical tasks. Here we propose the first-of-its-kind medical multimodal-multitask foundation model (M3FM) with application in lung cancer screening and related tasks. After we curated a comprehensive multimodal multitask dataset consisting of 49 clinical data types including 163,725 chest CT series and 17 medical tasks involved in LCS, we develop a multimodal question-answering framework as a unified training and inference strategy to synergize multimodal information and perform multiple tasks via free-text prompting. M3FM consistently outperforms the state-of-the-art single-modal task-specific models, identifies multimodal data elements informative for clinical tasks and flexibly adapts to new tasks with a small out-of-distribution dataset. As a specialty-oriented generalist medical AI model, M3FM paves the way for similar breakthroughs in other areas of medicine, closing the gap between specialists and the generalist.
BibTeX
@misc{niu2023specialty,
  author = {Chuang Niu and Qing Lyu and Christopher D. Carothers and Parisa Kaviani and Josh Tan and Pingkun Yan and Mannudeep K. Kalra and Christopher T. Whitlow and Ge Wang},
  title = {{Specialty-Oriented Generalist Medical AI for Chest CT Screening}},
  year = {2023},
  eprint = {2304.02649},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.02649},
}
Figure