Large language models are less effective at clinical prediction tasks than locally trained machine learning models

Katherine E. Brown, Chao Yan, Zhuohang Li, Xinmeng Zhang, Benjamin X. Collins, You Chen, Ellen Wright Clayton, Murat Kantarcioglu, Yevgeniy Vorobeychik, Bradley A. Malin

TL;DR

This study found that current large language models (LLMs) like GPT-3.5 and GPT-4 are significantly less effective than traditional, locally trained machine learning (ML) models for clinical prediction tasks. While LLMs show promise in fairness, their poor performance and calibration make them unreliable for direct clinical use without further development.

What Problem Does This Paper Address?

The rapid advancement and accessibility of LLMs have sparked interest in their potential to democratize AI in healthcare, particularly for institutions lacking the resources for traditional ML model development. However, the effectiveness, reliability, and fairness of these LLMs in real-world clinical prediction tasks remain largely unquantified. This paper addresses the critical question of whether LLMs can serve as viable substitutes for established ML models in healthcare, considering factors like predictive accuracy, calibration, privacy implications, and algorithmic fairness.

Methods

The researchers compared the performance of three LLMs (GPT-3.5, GPT-4, and GPT-4 with retrieval-augmented generation - RAG) against traditional gradient-boosting tree (GBT) models. They evaluated these models on two key clinical prediction tasks: predicting hospital discharge within 24 hours and predicting transfer to the ICU within 24 hours.

Key methodological aspects include:

  • Performance Measurement: Area Under the Receiver Operating Characteristic (AUROC) curve was used to assess predictive performance.
  • Calibration: Brier Score and calibration curves were employed to evaluate how well the predicted probabilities matched actual outcomes.
  • Privacy-Utility Tradeoff: The impact of data generalization and suppression (simulating privacy-preserving techniques) on model performance was assessed.
  • Algorithmic Fairness: Metrics like Average Absolute Odds Difference (AAOD) and Statistical Parity Difference (SPD) were used to evaluate fairness across demographic subgroups (race, sex, age).
  • In-Context Learning (RAG): The study explored the use of Retrieval Augmented Generation (RAG) to provide LLMs with relevant examples within prompts, aiming to improve their performance.

Datasets

The study utilized two primary electronic health record (EHR) derived datasets:

  • Vanderbilt University Medical Center (VUMC) dataset: Used to predict the likelihood of patient discharge from the hospital within 24 hours.
  • MIMIC-IV and MIMIC-IV ED datasets: Publicly available datasets from Beth Israel Deaconess Medical Center, used to predict the likelihood of transfer to the ICU within 24 hours after emergency department triage.

Key Insights and Ideas

  • LLMs Lag Behind Traditional ML in Predictive Power: The most striking finding is the substantial performance gap between LLMs and traditional ML models. This suggests that LLMs, in their current non-fine-tuned state, lack the discriminative capabilities for these specific clinical prediction tasks.
  • Calibration is a Major Hurdle for LLMs: LLMs exhibited poor calibration, meaning their predicted probabilities did not accurately reflect the likelihood of the actual outcome. This unreliability is a significant concern for clinical decision-making.
  • Fairness vs. Performance Trade-off: While LLMs, particularly GPT-4, showed better fairness metrics in some scenarios, this often came at the cost of significantly reduced predictive performance. This highlights a complex trade-off that needs careful consideration.
  • RAG's Limited Impact: Retrieval-augmented generation (RAG) did not consistently improve LLM performance or fairness, suggesting that simply providing examples might not be sufficient without more sophisticated fine-tuning or prompt engineering.
  • LLMs' Sensitivity to Data Generalization: LLMs were more sensitive to the generalization and suppression of demographic information compared to traditional ML, indicating a potential reliance on these features for prediction. This raises questions about how privacy-preserving techniques might impact LLM utility.

Key Results

  • AUROC: Traditional ML models achieved significantly higher AUROC scores (e.g., 0.847 for VUMC, 0.894 for MIMIC-IV) compared to GPT-3.5 (e.g., 0.537 for VUMC, 0.517 for MIMIC-IV) and GPT-4 (e.g., 0.629 for VUMC, 0.602 for MIMIC-IV).
  • Brier Score: Traditional ML models had substantially lower Brier Scores (indicating better calibration), with scores like 0.134 (VUMC) and 0.042 (MIMIC-IV), compared to GPT-3.5 (0.384 and 0.06) and GPT-4 (0.251 and 0.219).
  • Fairness: GPT-4 was found to be the fairest model according to selected metrics, but this was achieved with lower performance. Traditional ML was not always the least fair model.
  • Privacy: Traditional ML performance remained largely consistent across different levels of demographic data generalization, while LLMs showed more variability.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Scope: The study focused on only two clinical prediction tasks and data from two medical centers, which may limit the generalizability of the findings.
  • Interpretability Not Assessed: The study did not evaluate interpretability, which is a crucial aspect for clinical AI adoption.
  • Operational Costs Ignored: The significant costs associated with implementing and maintaining LLMs were not considered.
  • Privacy-Utility Tradeoff Nuances: The authors noted that the privacy-utility tradeoff observed might not perfectly reflect expectations, suggesting potential confounding factors in the datasets.

Future work should focus on:

  • Optimizing In-Context Learning: Developing more effective strategies for in-context learning and RAG to improve LLM performance.
  • Randomization-Based Privacy: Investigating the impact of randomization-based privacy preservation strategies.
  • Internal Biases: Understanding and mitigating the internal biases within LLMs.
  • Broader Evaluation: Conducting studies across a wider variety of clinical tasks and medical centers to assess generalizability.
  • Fine-tuning: While not the focus of this study, fine-tuning LLMs with local data is a critical area for future research to improve their performance in clinical settings.

Summary updated

Abstract

To determine the extent to which current large language models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity.
BibTeX
@article{brown2025large,
  author = {Katherine E. Brown and Chao Yan and Zhuohang Li and Xinmeng Zhang and Benjamin X. Collins and You Chen and Ellen Wright Clayton and Murat Kantarcioglu and Yevgeniy Vorobeychik and Bradley A. Malin},
  title = {{Large language models are less effective at clinical prediction tasks than locally trained machine learning models}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2025},
  doi = {10.1093/jamia/ocaf038},
}
Figure