Outlier detection for patient monitoring and alerting

Milos Hauskrecht, Iyad Batal, Michal Valko, Shyam Visweswaran, Gregory F. Cooper, Gilles Clermont

TL;DR

This paper introduces a data-driven method for detecting potential medical errors by identifying unusual patient management decisions within electronic health records (EHRs). By learning from historical patient data, their system flags actions that deviate significantly from typical care, aiming to provide timely alerts to clinicians.

What Problem Does This Paper Address?

Medical errors remain a significant problem in healthcare, leading to patient harm and increased costs. Current systems for alerting clinicians to potential errors are often knowledge-based, requiring extensive manual effort to build and maintain, and can have limited coverage. This leaves a gap for more comprehensive and adaptable alerting systems. The authors propose to address this by leveraging the vast amount of data in EHRs to identify "outlier" patient management decisions that might indicate an error, without relying on pre-defined clinical rules.

Methods

The core of their approach is conditional outlier detection. Instead of just looking for data points that are unusual in isolation, they identify patient management actions (like ordering a medication or a lab test) that are unusual given the patient's current condition.

The process involves two main stages:

  1. Model Building:
    • Segmentation: EHR data is segmented into discrete time intervals, creating "patient states" at each point.
  2. Feature Engineering: Patient states are represented by a rich set of features derived from temporal patterns of lab tests, medications, and procedures. This includes things like the difference between recent lab values, time since last order, and indicators of active medications.
  3. Probabilistic Modeling: For each type of patient management action (e.g., ordering a specific lab test), a predictive model is built. This model learns the probability of that action occurring given a patient's state. Support Vector Machines (SVMs) with a linear kernel are used, and a non-parametric method is employed to estimate class posteriors. Feature selection is done greedily in groups to manage the high dimensionality of the data.
  4. Alert Generation:
    • When a new patient action is observed, its "anomaly score" is calculated. This score quantifies how unusual the action is given the patient's state.
  5. An "alert score" is then computed, which considers the anomaly score for the current patient state and the previous one, ensuring that a surprising action is consistently surprising over a short period.
  6. If this alert score exceeds a predefined threshold, an alert is triggered for the clinician.

Datasets

The study utilized a dataset of 4486 post-cardiac surgical patients from archived EHRs at a large teaching hospital. This dataset was split into a training set (2878 cases from 2002-2004) and a test set (1608 cases from 2005-2006). The segmentation process resulted in over 51,000 patient-state instances.

Key Insights and Ideas

  • Context is King: The most crucial insight is that an action's "unusualness" is highly dependent on the patient's context. A seemingly odd decision might be perfectly reasonable in a specific clinical scenario. Their conditional approach captures this.
  • Data-Driven Discovery: The method bypasses the need for manual rule creation by experts, instead learning patterns directly from historical data. This allows for broader coverage and adaptation to evolving clinical practices.
  • Anomaly Score as a Proxy for Error: The core hypothesis is that statistically unusual patient management actions are good indicators of potential medical errors. The evaluation aims to validate this link.
  • Alert Score for Urgency: The alert score mechanism provides a way to prioritize alerts, focusing on actions that are not only statistically anomalous but also persistently so.

Key Results

  • The outlier-based alerting approach demonstrated promising true alert rates, ranging from 25% to 66%.
  • The highest true alert rates (66%) were observed for the strongest outliers (those with the highest alert scores).
  • Expert review of 222 generated alerts indicated that a significant portion were considered useful. For "Item 1" (usefulness of the alert), the true alert rate was 55%, and for "Item 2" (likelihood of following up), it was 51%.
  • For "stronger" alerts (those with alert scores in the [0.8, 1.0] range), the true alert rates increased to 66% for Item 1 and 64% for Item 2.
  • The study found a positive correlation between alert scores and true alert rates, suggesting that higher anomaly scores are more likely to correspond to clinically useful alerts.

Limitations and Future Work

  • Offline Evaluation: The study was conducted offline on retrospective data, not accounting for real-world factors like alert fatigue, workflow interruptions, or the impact of alert prioritization.
  • Limited Alert Scope: To manage the evaluation, the number of alerts considered for each action was capped, potentially limiting the insights into weaker anomalies.
  • Individual Action Models: Current models focus on individual patient management actions, not multivariate relationships. This could lead to missing complex outliers where multiple actions interact.
  • Segmentation Resolution: The fixed 24-hour segmentation might not be optimal for actions with shorter time horizons.
  • Feature Set Limitations: While effective for many actions, the current feature sets did not yield good predictive models for all.

Future work could involve:

  • Developing multivariate outlier detection models.
  • Investigating more flexible time segmentation strategies.
  • Exploring new feature sets to better characterize complex time-series data.
  • Integrating this system into real-world clinical workflows to assess its impact on alert fatigue and override rates.

Summary updated

Abstract

We develop and evaluate a data-driven approach for detecting unusual (anomalous) patient-management decisions using past patient cases stored in electronic health records (EHRs). Our hypothesis is that a patient-management decision that is unusual with respect to past patient care may be due to an error and that it is worthwhile to generate an alert if such a decision is encountered. We evaluate this hypothesis using data obtained from EHRs of 4486 post-cardiac surgical patients and a subset of 222 alerts generated from the data. We base the evaluation on the opinions of a panel of experts. The results of the study support our hypothesis that the outlier-based alerting can lead to promising true alert rates. We observed true alert rates that ranged from 25% to 66% for a variety of patient-management actions, with 66% corresponding to the strongest outliers.
BibTeX
@article{hauskrecht2013outlier,
  author = {Milos Hauskrecht and Iyad Batal and Michal Valko and Shyam Visweswaran and Gregory F. Cooper and Gilles Clermont},
  title = {{Outlier detection for patient monitoring and alerting}},
  journal = {J. Biomed. Informatics},
  year = {2013},
  url = {https://openreview.net/forum?id=YeylxnJPv5A},
}
Figure