Stable Prediction of Adverse Events in Medical Time-Series Data

Abstract

Early Event Prediction (EEP) systems in healthcare are designed to provide clinicians with early warnings for critical conditions like sepsis. However, many current models suffer from "alert volatility," where risk scores fluctuate erratically over short intervals. We introduce CAREBench, a new multi-modal benchmark that evaluates models on both snapshot accuracy and temporal stability. Our evaluation across six clinical tasks reveals that while deep learning and Large Language Models (LLMs) offer high predictive power, they often exhibit significant instability compared to classical methods.

Introduction: The "Jitter" Problem

Clinical trust in AI monitoring systems is often undermined by the "jitter" problem. Most benchmarks evaluate models using "per-window" accuracy, ignoring the temporal consistency of predictions. A model may be correct on average but produce risk scores that oscillate wildly between 10% and 90% within minutes. This volatility leads to alarm fatigue and a lack of reliability in real-world bedside settings.

Expand

Source, p. 3

CAREBench: A Multi-modal Stability Benchmark

To address these limitations, we developed CAREBench. Unlike previous benchmarks that focus primarily on tabular Electronic Health Record (EHR) data, CAREBench integrates continuous physiologic waveforms and unstructured clinical notes. Crucially, it shifts evaluation from simple snapshots to per-patient longitudinal stability.

Expand

Source, p. 2

The benchmark utilizes three major datasets: MC-MED (high-resolution waveforms), MIMIC-IV (EHR and text), and EHRShot (longitudinal tabular data). Stability is quantified using a modified local Lipschitz constant, measuring the rate of change in risk scores relative to time.

Methods and Model Architectures

We compared several model classes:

• Classical Baselines: XGBoost and Random Forests using summary statistics.
• Deep Sequence Models: Mamba and GPT-2 processing raw data sequences.
• Zero-shot LLMs: Qwen3-32B utilizing structured prompting and chain-of-thought reasoning.

To prevent models from learning simple heuristics (e.g., longer stays implying sicker patients), we implemented propensity-based sampling to balance history lengths across cases.

Key Insights and Results

The Stability-Accuracy Trade-off

Our results indicate a clear trade-off between performance and reliability. While deep learning models like GPT-2 often achieved the highest AUROC, they were notably less stable than tree-based models. However, an auto-regressive approach with GPT-2—predicting the next state before the risk—acted as a natural stabilizer, producing significantly smoother trajectories.

LLM Performance and Volatility

Zero-shot LLMs demonstrated impressive reasoning, particularly on text-heavy tasks like in-hospital mortality (AUROC 0.86). However, they were the most "jittery" models tested. In the hyperkalemia task, Qwen3-32B's stability score was 10.67, compared to near-zero for auto-regressive GPT-2. This suggests that while LLMs can synthesize complex notes, every new token of data can trigger a vastly different risk estimate.

Conclusion

CAREBench demonstrates that high accuracy is insufficient for clinical deployment. For AI to earn clinician trust, it must provide stable, consistent risk trajectories. Future research must focus on developing architectures that maintain the reasoning power of LLMs and deep sequence models while enforcing the temporal stability required for safe bedside monitoring.

@misc{keoliya2025stable,
  author = {Mayank Keoliya and Seewon Choi and Rajeev Alur and Mayur Naik and Eric Wong},
  title = {{Stable Prediction of Adverse Events in Medical Time-Series Data}},
  year = {2025},
  eprint = {2510.14286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14286},
}