Leveraging LLMs for Predicting Unknown Diagnoses from Clinical Notes

Abstract

Electronic Health Records (EHRs) frequently fail to explicitly link prescribed medications to their underlying diagnoses, which hinders clinical decision support and predictive analytics. This study investigates the capability of Large Language Models (LLMs) to deduce these implicit links from unstructured clinical notes. Using a zero-shot prompting approach with GPT-3.5 Turbo, the researchers developed an ensemble method that varies hyperparameter configurations—such as temperature, top-p, and summary length—and combines results via majority voting. The findings demonstrate that an ensemble of diverse configurations significantly outperforms individual models, increasing prediction accuracy from 66% to 75% on a newly created expert-annotated benchmark.

Introduction

In modern healthcare, Electronic Health Records (EHRs) are often incomplete. While they list medications and diagnoses, the causal link between them is rarely structured. Discharge summaries contain narrative explanations of these relationships, but extracting them reliably is a challenge. This research explores whether LLMs can bridge this gap by reading unstructured notes to identify the specific diagnoses justifying medication prescriptions.

Methods

The study utilized a zero-shot prompting strategy with GPT-3.5 Turbo. The core innovation is an ensemble approach designed to capture a wider range of linguistic patterns than a single optimized run.

Expand

Source, p. 6

The researchers systematically varied three parameters to create 18 distinct configurations:

• Temperature: 0.1 (deterministic) to 0.95 (exploratory).
• Top-p: 0.1 (narrow) to 0.9 (broad).
• Summary Length: Short (~2000 characters) vs. Long (~4000 characters).

From these 18 configurations, random subsets of five were selected for majority voting. The dataset was derived from MIMIC-IV clinical notes, with 240 medication-diagnosis pairs manually annotated by experts to serve as a ground-truth benchmark.

Results

The experimental results highlight the superiority of ensemble methods over single-model configurations. The best-performing single run achieved 66% accuracy, while the optimized ensemble reached 75%.

Key insights include:

• Diversity Matters: The most effective ensembles combined deterministic, balanced, and exploratory settings. This variety allows the model to capture complementary information.
• Context Length Paradox: Shorter summaries (~2000 characters) generally outperformed longer ones (~4000 characters), likely because they focused the LLM's attention on relevant details and avoided the "lost in the middle" phenomenon.
• Noise Sensitivity: High creativity settings (high temperature) combined with long context lengths led to performance degradation, suggesting that increased variability introduces noise when processing extended clinical text.

Discussion and Limitations

While promising, the study is limited by its small dataset size (240 pairs) due to the difficulty of expert annotation. Additionally, the research focused solely on GPT-3.5 Turbo. Future work should explore more advanced models like GPT-4 or Med-PaLM 2, larger datasets, and more sophisticated ensemble techniques such as weighted voting or stacking.

Conclusion

Strategic ensembling of LLMs with diverse hyperparameter settings offers a robust pathway for extracting structured clinical insights from unstructured data, significantly improving the reliability of automated diagnostic prediction in EHR systems.

@misc{albassam2025leveraging,
  author = {Dina Albassam and Adam Cross and Chengxiang Zhai},
  title = {{Leveraging LLMs for Predicting Unknown Diagnoses from Clinical Notes}},
  year = {2025},
  eprint = {2503.22092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.22092},
}