Large language models are less effective at clinical prediction tasks than locally trained machine learning models

Katherine E. Brown, Chao Yan, Zhuohang Li, Xinmeng Zhang, Benjamin X. Collins, You Chen, Ellen Wright Clayton, Murat Kantarcioglu, Yevgeniy Vorobeychik, Bradley A. Malin

Abstract

To determine the extent to which current large language models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity.
BibTeX
@article{brown2025large,
  author = {Katherine E. Brown and Chao Yan and Zhuohang Li and Xinmeng Zhang and Benjamin X. Collins and You Chen and Ellen Wright Clayton and Murat Kantarcioglu and Yevgeniy Vorobeychik and Bradley A. Malin},
  title = {{Large language models are less effective at clinical prediction tasks than locally trained machine learning models}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2025},
  doi = {10.1093/jamia/ocaf038},
}

Loading related content...

Figure