Large AI Models in Health Informatics: Applications, Challenges, and the Future

Jianing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P. -w. Lo, Bo Xiao, Wu Yuan, Ningli Wang, Dong Xu, Benny Lo

TL;DR

This paper provides a comprehensive roadmap of how Large AI Models (LAMs)—such as ChatGPT and the Segment Anything Model (SAM)—are fundamentally shifting the landscape of health informatics. By reviewing applications across seven key sectors, from bioinformatics to medical robotics, the authors highlight a transition from task-specific "narrow" AI to versatile "foundation models" that promise to augment clinical decision-making while facing significant hurdles in reliability, privacy, and ethics.

What Problem Does This Paper Address?

Traditional medical AI has long been plagued by the "bottleneck of supervision." Developing a model for a specific task, like detecting a particular tumor in a CT scan, typically requires thousands of high-quality images meticulously annotated by expensive domain experts. These models are often brittle, failing when applied to slightly different data or new tasks.

The authors address how the rise of LAMs offers a way out. These models are pre-trained on massive, multi-modal datasets using self-supervision, allowing them to develop a "generalist intelligence." This enables them to perform remarkably well on new medical tasks with little to no additional training (zero-shot or few-shot learning), potentially solving the scalability and generalization issues that have hindered medical AI for decades.

Methods

The paper is a systematic review that categorizes the current state of LAMs in healthcare. The authors define a three-part taxonomy for these models:

  1. Large Language Models (LLMs): Pre-trained on text (e.g., GPT-4, PaLM 2) to handle medical Q&A and clinical documentation.
  2. Large Vision Models (LVMs): Pre-trained on images (e.g., SAM) for medical imaging and diagnostics.
  3. Large Multi-modal Models (LMMs): Models that integrate multiple data types (text, images, signals) to provide a more holistic clinical view.

The core methodology focuses on identifying the paradigm shift (illustrated in Figure 2) from limited-scale, task-specific models to large-scale foundation models capable of broad generalization.

The paradigm shift from narrow AI to Large AI Models.

Datasets

The authors emphasize that while general-domain datasets are massive (trillions of tokens), medical datasets are catching up but remain smaller due to privacy and curation costs. Key datasets mentioned include:

  • Bioinformatics: Big Fantastic Database (2.1B protein sequences) and RNAcentral.
  • Medical Diagnosis: MIMIC-CXR (337k chest X-rays) and Mount Sinai ECG Data (8.5M recordings).
  • Medical Informatics: Medical Meadow (1.5M data points for NLP tasks).
  • Medical Robotics: Endo-FM database (5M endoscopic frames).

Key Insights and Ideas

  • The Multi-modal Advantage: Healthcare is inherently multi-modal. A doctor doesn't just look at an X-ray; they read the patient's history and check lab results. LAMs are uniquely suited to this because they can be trained to "align" different data types into a single representational space.
  • Emergent Reasoning: As models scale to billions of parameters, they develop "emergent" abilities like complex clinical reasoning that weren't explicitly programmed. For example, GPT-4 can act as a "Socratic tutor" in medical education, guiding students through problems rather than just giving answers.
  • Speed vs. Accuracy in Bioinformatics: While experimental methods like Cryo-EM are the gold standard for protein structure, LAMs like ESMfold can make predictions several orders of magnitude faster, which is a game-changer for rapid drug discovery.
  • The Research Explosion: The release of ChatGPT and SAM triggered an unprecedented surge in medical AI research, with hundreds of publications appearing monthly within just a year of their launch (see Figure 1).
Fig. 1: Number of publications related to ChatGPT and SAM in medical and health areas. Statistics were queried from Google Scholar with the keywords “Medical ChatGPT” or “Medical Segment Anything”, and the last entry was 31-th Aug. 2023. From April to August, each month, there were over 200 publications about ChatGPT in medicine and healthcare.

Key Results

The paper summarizes impressive benchmarks where LAMs outperform previous state-of-the-art (SOTA) methods:

  • Medical Q&A: Med-PaLM 2 achieved 86.5% accuracy on the MedQA (USMLE-style) benchmark.
  • Protein Folding: ESMfold achieved a TM-score of 0.68 on CASP14, significantly outperforming AlphaFold2 (0.38) when evolutionary information (MSAs) is unavailable.
  • Medical Imaging: STU-Net, a 1.4B parameter model, reached 90.1% Dice score on the TotalSegmentator dataset, setting a new bar for medical image segmentation.
  • Education: GPT-4 passed the USMLE sample exam with 86.7% accuracy, demonstrating expert-level medical knowledge.

Limitations and Future Work

Despite the promise, the authors identify several "red flags":

  • Reliability & Hallucination: LLMs can generate factually incorrect but highly plausible medical advice, which is dangerous in clinical settings.
  • Data Privacy: LAMs have a high capacity to memorize training data, leading to risks of leaking sensitive patient information through "jailbreaking" prompts.
  • Bias and Fairness: If training data is biased (e.g., underrepresenting certain ethnicities), the model's diagnostic performance will be lower for those groups.
  • Interpretability: These models are "black boxes." For a doctor to trust an AI's diagnosis, they need to understand why it made that choice—a capability LAMs currently lack.

Future Directions: The authors call for the development of "World Models" for health that integrate all biological scales (from molecules to populations) and the creation of robust regulatory frameworks to ensure these models are deployed safely and ethically.

Summary updated

Abstract

Large AI models, or foundation models, are models recently emerging with massive scales both parameter-wise and data-wise, the magnitudes of which can reach beyond billions. Once pretrained, large AI models demonstrate impressive performance in various downstream tasks. A prime example is ChatGPT, whose capability has compelled people's imagination about the far-reaching influence that large AI models can have and their potential to transform different domains of our lives. In health informatics, the advent of large AI models has brought new paradigms for the design of methodologies. The scale of multi-modal data in the biomedical and health domain has been ever-expanding especially since the community embraced the era of deep learning, which provides the ground to develop, validate, and advance large AI models for breakthroughs in health-related areas. This article presents a comprehensive review of large AI models, from background to their applications. We identify seven key sectors in which large AI models are applicable and might have substantial influence, including 1) bioinformatics; 2) medical diagnosis; 3) medical imaging; 4) medical informatics; 5) medical education; 6) public health; and 7) medical robotics. We examine their challenges, followed by a critical discussion about potential future directions and pitfalls of large AI models in transforming the field of health informatics.
BibTeX
@article{qiu2023large,
  author = {Jianing Qiu and Lin Li and Jiankai Sun and Jiachuan Peng and Peilun Shi and Ruiyang Zhang and Yinzhao Dong and Kyle Lam and Frank P. -W. Lo and Bo Xiao and Wu Yuan and Ningli Wang and Dong Xu and Benny Lo},
  title = {{Large AI Models in Health Informatics: Applications, Challenges, and the Future}},
  journal = {IEEE J. Biomed. Health Informatics},
  year = {2023},
  url = {https://openreview.net/forum?id=5NqrtP5DgR},
}
Figure