Monitoring Deployed AI Systems in Health Care

Timothy Keyes, Alison Callahan, Abby S. Pandya, Nerissa Ambers, Juan M. Banda, Miguel Fuentes, Carlene Lugtu, Pranav Masariya, Srikar Nallan, Connor O'Brien, Thomas Wang, Emily Alsentzer, Jonathan H. Chen, Dev Dash, Matthew A. Eisenberg, Patricia Garcia, Nikesh Kotecha, Anurang Revri, Michael A. Pfeffer, Nigam H. Shah, Sneha S. Jain

TL;DR

Stanford Health Care researchers have developed a practical, action-oriented framework for monitoring deployed AI systems in clinical settings, organized around three pillars: system integrity, performance, and impact. Tested across 13 active deployments, this framework bridges the gap between high-level AI governance and day-to-day operational decision-making, providing a clear blueprint for when to retrain, reconfigure, or decommission clinical models.

What Problem Does This Paper Address?

Deploying an AI model in a hospital isn't a one-and-done milestone; it's an ongoing operational commitment. Clinical IT environments are notoriously dynamic—electronic health record (EHR) upgrades routinely alter underlying data schemas, and clinical practices shift over time, leading to "dataset shift" or "concept drift." When data pipelines break or model accuracy degrades silently, patient safety is put at risk.

While many healthcare organizations have proposed high-level AI governance guidelines, there is a distinct lack of pragmatic, operational frameworks explaining how to monitor these systems, who should look at the dashboards, and what concrete actions to take when a model begins to fail.

Methods

The authors introduce a technology-agnostic monitoring framework integrated into Stanford Health Care’s "Responsible AI Lifecycle" (RAIL) process. The framework evaluates AI systems across three complementary dimensions:

  1. System Integrity Monitoring: Focuses on the technical plumbing. It tracks service uptime, API request latency, and data retrieval failures (e.g., missing features or API timeouts) in real time.
  2. Performance Monitoring: Evaluates the statistical accuracy and quality of outputs over time (e.g., AUROC, precision, recall). Crucially, it incorporates equity by requiring subgroup-stratified metrics to detect differential performance across patient populations.
  3. Impact Monitoring: Assesses downstream clinical and operational utility. It answers whether the AI is actually delivering value, tracking metrics like user adoption rates, KPI improvements (time or labor savings), and surveillance for unintended harms (e.g., spikes in rapid response calls).
[AI System Deployed] 
       │
       ├──> System Integrity ──> Real-time On-Call Support / Error Logging
       ├──> Performance      ──> Statistical Dashboards / Subgroup Audits
       └──> Impact           ──> Process & Outcome KPIs / User Feedback
       │
       └──> [Governance Committee Review] ──> Retrain, Reconfigure, or Retire

The framework explicitly tailors its strategy based on the AI type:

  • Traditional AI: Monitored via standard statistical classification metrics.
  • Generative AI (Fixed-Prompt): Automated batch systems (e.g., screening patients for palliative care) are monitored by linking outputs to downstream clinical adjudication data in the EHR.
  • Generative AI (Open-Prompt): Interactive tools (e.g., clinical chatbots) use usage-informed benchmarking, log analysis, and real-time safety guardrails since exhaustive benchmarking is impossible.

Datasets

The paper does not introduce a new public machine learning dataset. Instead, it details the operationalization of live, institutional data streams within Stanford Health Care.

For generative AI evaluation, the authors highlight MedHELM, an internally developed framework that ingests gold-standard, human-labeled benchmark datasets assembled via manual chart abstraction. These living benchmarks are used to run scheduled batch evaluations across changing LLM versions.

Key Insights and Ideas

  • Actionability is Pre-Specified: Monitoring is useless without a plan. Before any model is deployed, alert thresholds and exact follow-up actions (e.g., routing a ticket to a specific DevOps engineer or clinical owner) must be defined.
  • Equity is Embedded, Not Siloed: Rather than treating equity as a standalone fourth monitoring domain, the authors integrate it directly into performance and impact tracking via subgroup analysis.
  • Countering the "Deployment Bias": In IT infrastructure, the number of active systems is often viewed as a success metric. This framework actively counters that culture by establishing formal pathways to decommission low-value or unmonitorable models.
  • Two-Tiered Cadence: The authors advocate for pairing real-time technical support (for acute system integrity failures) with periodic, longitudinal governance reviews (monthly/quarterly for performance and impact drift) to avoid alert fatigue caused by normal statistical noise.

Key Results

The framework has been successfully operationalized for 13 active deployments (6 traditional AI, 7 generative AI) and 4 planned deployments at Stanford Health Care. Real-world outcomes of this monitoring include:

  • Expansion: A 90-day post-go-live review of an LLM-powered inpatient hospice screen validated its utility, supporting its expansion from a 2-unit pilot to all inpatient units.
  • Workflow Optimization: Impact reviews of a Peripheral Artery Disease (PAD) classifier revealed low process metrics, prompting successful workflow modifications to improve provider notification and workup rates.
  • Model Retirement: A systematic review of six Epic-developed traditional models resulted in decommissioning four models (including Risk of Inpatient Falls and ICU Length of Stay) because they were not tightly integrated into clinical workflows and therefore could not support impact monitoring. Two other models were retired and replaced with superior versions.

Limitations and Future Work

  • Resource Intensity: Sustaining this level of comprehensive monitoring requires a dedicated, multi-disciplinary team of data engineers, scientists, and clinical informaticians—a luxury many resource-constrained health systems cannot afford.
  • The Third-Party Black Box: Monitoring commercial, vendor-served AI remains difficult due to limited visibility into proprietary telemetry. Health systems may need to mandate minimal monitoring capabilities contractually during procurement.
  • Heuristic Thresholds: The current default performance alerting threshold (a 75–125% acceptance band relative to validation) is a simple heuristic. The authors plan to transition to more rigorous Statistical Process Control (SPC) methodologies as more data accrues.
  • LLM Evaluation Bottlenecks: Relying on manual chart abstraction for gold-standard LLM evaluation is labor-intensive. Future work will explore semi-automated evaluation corpus construction and the cautious use of "LLM-as-a-judge" silver standards.

Summary updated

Abstract

Post-deployment monitoring of artificial intelligence (AI) systems in health care is essential to ensure their safety, quality, and sustained benefit-and to support governance decisions about which systems to update, modify, or decommission. Motivated by these needs, we developed a framework for monitoring deployed AI systems grounded in the mandate to take specific actions when they fail to behave as intended. This framework, which is now actively used at Stanford Health Care, is organized around three complementary principles: system integrity, performance, and impact. System integrity monitoring focuses on maximizing system uptime, detecting runtime errors, and identifying when changes to the surrounding IT ecosystem have unintended effects. Performance monitoring focuses on maintaining accurate system behavior in the face of changing health care practices (and thus input data) over time. Impact monitoring assesses whether a deployed system continues to have value in the form of benefit to clinicians and patients. Drawing on examples of deployed AI systems at our academic medical center, we provide practical guidance for creating monitoring plans based on these principles that specify which metrics to measure, when those metrics should be reviewed, who is responsible for acting when metrics change, and what concrete follow-up actions should be taken-for both traditional and generative AI. We also discuss challenges to implementing this framework, including the effort and cost of monitoring for health systems with limited resources and the difficulty of incorporating data-driven monitoring practices into complex organizations where conflicting priorities and definitions of success often coexist. This framework offers a practical template and starting point for health systems seeking to ensure that AI deployments remain safe and effective over time.
BibTeX
@misc{keyes2025monitoring,
  author = {Timothy Keyes and Alison Callahan and Abby S. Pandya and Nerissa Ambers and Juan M. Banda and Miguel Fuentes and Carlene Lugtu and Pranav Masariya and Srikar Nallan and Connor O'Brien and Thomas Wang and Emily Alsentzer and Jonathan H. Chen and Dev Dash and Matthew A. Eisenberg and Patricia Garcia and Nikesh Kotecha and Anurang Revri and Michael A. Pfeffer and Nigam H. Shah and Sneha S. Jain},
  title = {{Monitoring Deployed AI Systems in Health Care}},
  year = {2025},
  eprint = {2512.09048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.09048},
}
Figure