Abstract
Large AI models, or foundation models, are models recently emerging with massive scales both parameter-wise and data-wise, the magnitudes of which often reach beyond billions. Once pretrained, large AI models demonstrate impressive performance in various downstream tasks. A concrete example is the recent debut of ChatGPT, whose capability has compelled people's imagination about the far-reaching influence that large AI models can have and their potential to transform different domains of our life. In health informatics, the advent of large AI models has brought new paradigms for the design of methodologies. The scale of multimodality data in the biomedical and health domain has been ever-expanding especially since the community embraced the era of deep learning, which provides the ground to develop, validate, and advance large AI models for breakthroughs in health-related areas. This article presents an up-to-date comprehensive review of large AI models, from background to their applications. We identify seven key sectors that large AI models are applicable and might have substantial influence, including 1) molecular biology and drug discovery; 2) medical diagnosis and decision-making; 3) medical imaging and vision; 4) medical informatics; 5) medical education; 6) public health; and 7) medical robotics. We examine their challenges in health informatics, followed by a critical discussion about potential future directions and pitfalls of large AI models in transforming the field of health informatics.
Introduction
Traditionally, medical AI followed a "small and supervised" paradigm: researchers trained specific models on meticulously labeled datasets for single tasks, such as detecting pneumonia in chest X-rays. While effective for narrow applications, this approach is expensive, scales poorly, and fails to capture the multi-modal nature of health data, where a single patient might have genomic sequences, clinical notes, and 3D imaging simultaneously.
The landscape changed dramatically with the introduction of Large AI Models (LAMs). As shown in the figure below, the release of general-purpose models like ChatGPT and the Segment Anything Model (SAM) triggered an explosion of research interest in the medical domain, as researchers sought to adapt these generalist tools to specialized clinical needs.
The Paradigm Shift in Health Informatics
The core transition in the field is the move from "Engineer-in-the-loop"—where experts must hand-craft features for specific tasks—to "User-in-the-loop," where clinicians interact with generalist models via natural language prompts. This shift is characterized by four primary dimensions: increasing model size (reaching hundreds of billions of parameters), expanding data scale, the integration of multiple modalities (text, image, audio, etc.), and enhanced versatility across downstream tasks.
Categorizing Large AI Models
LAMs can be broadly categorized into three primary types:
- Large Language Models (LLMs): Pre-trained on vast text corpora to handle tasks like medical question answering and clinical note summarization.
- Large Vision Models (LVMs): Scaled-up architectures like Vision Transformers (ViTs) used for zero-shot medical image segmentation.
- Large Multi-modal Models (LMMs): Models that bridge the gap between text and vision, enabling "visual chat" where a doctor can query a model about specific pathology slides or X-rays.
Key Insights and Results
Scaling these models does more than just improve accuracy; it unlocks "emergent" abilities. For instance, Med-PaLM 2 achieved an impressive 86.5% on the MedQA (USMLE-style) benchmark, demonstrating a capacity for nuanced medical reasoning rather than simple fact retrieval.
In bioinformatics, treating biological sequences (DNA, RNA, proteins) as a language has allowed models like ESMfold (15B parameters) to predict 3D protein structures orders of magnitude faster than traditional methods, achieving a TM-score of 0.68. Similarly, in medical imaging, models like STU-Net (1.4B parameters) have shown that scaling vision architectures directly correlates with superior performance in large-scale segmentation tasks.
Challenges and Future Directions
Despite their potential, LAMs face critical pitfalls in clinical settings:
- Reliability & Hallucination: Models can generate factually incorrect yet plausible content, posing significant risks to patient safety.
- Data Privacy: The "excessive capacity to memorize" in large models creates risks that sensitive patient data could be extracted through adversarial prompting.
- Fairness & Bias: Training data biased toward specific demographics or languages can lead to inequitable clinical utility.
The ultimate goal for the field is the development of a "World Model" for healthcare—a single, unified foundation model that integrates all medical modalities (genomics, imaging, sensors, and text) to serve as a comprehensive, generalist clinical assistant.
Summary updated