Predicting postoperative delirium assessed by the Nursing Screening Delirium Scale in the recovery room for non-cardiac surgeries without craniotomy: A retrospective study using a machine learning approach

Niklas Giesa, Stefan Haufe, Mario Menk, Björn Weiß, Claudia Spies, Sophie K. Piper, Felix Balzer, Sebastian Daniel BoiemedRxiv 2023 · DOI

TL;DR

Researchers have developed non-linear machine learning models that significantly improve the prediction of postoperative delirium (POD) in the recovery room. By integrating data from across the entire surgical journey—preoperative, intraoperative, and immediate postoperative phases—their tree-based models achieved an AUROC of 0.854, far outperforming traditional linear methods and existing clinical baselines.

What Problem Does This Paper Address?

Postoperative delirium (POD) is an acute state of brain dysfunction that can lead to devastating long-term consequences, including permanent cognitive decline, dementia, and increased mortality. Despite its impact, predicting who will succumb to it remains a challenge.

Most existing prediction models suffer from two main flaws:

  1. Linearity: They rely on simple logistic regression, which struggles to capture the complex, non-linear interactions of physiological data.
  2. Static Snapshots: They often use "time-invariant" data (like age or pre-existing conditions) from a single point in time, ignoring the fluctuating conditions a patient experiences during and immediately after surgery.

This study aims to bridge this gap by using non-linear machine learning to predict POD specifically in the recovery room (PACU) setting, providing clinicians with a window for early intervention.

Methods

The researchers compared traditional linear logistic regression (LR) against two types of non-linear machine learning architectures:

  • Deep Multi-Layer Perceptrons (MLPs): Neural networks designed to find complex patterns in data.
  • Tree-Based Ensembles: Specifically Random Forest and Gradient Boosting classifiers.

A key novelty was the temporal structuring of the data into three phases:

  • T1 (Preoperative): Hospital admission to anesthesia start.
  • T2 (Intraoperative): The duration of the surgery.
  • T3 (Postoperative): Anesthesia end to the first delirium screening in the recovery room.

To handle the "needle in a haystack" problem—where most patients do not develop delirium—the team used specialized loss functions (Focal Loss and Weighted Binary Cross-Entropy) to prevent the models from simply ignoring the minority of POD-positive cases.

Datasets

The study utilized a massive retrospective dataset from Charité – Universitätsmedizin Berlin:

  • Scope: 73,181 surgeries across 61,187 distinct patients between 2017 and 2020.
  • Inclusion: Adult patients (18+) undergoing non-cardiac surgeries without craniotomy.
  • Target Variable: POD was defined as a score of ≥ 1 on the Nursing Screening Delirium Scale (Nu-DESC).
  • Features: 549 clinical variables were extracted from Electronic Health Records (EHRs), covering demographics, medications, and high-frequency physiological data.

Key Insights and Ideas

  • The Power of "When": The study found that models using data from all three phases (T1+T2+T3) performed the best. However, even models using only pre- and intraoperative data (T1+T2) were highly effective, which is crucial because it allows for a "heads-up" before the patient even reaches the recovery room.
  • Non-Linearity Wins: Tree-based models generally outperformed MLPs. The authors suggest this is because decision trees are naturally better at handling the specific feature sets and tabular data structures found in clinical records.
  • Fluctuations Matter: Traditional models often ignore intraoperative "chaos." This study showed that variables like the amount of remifentanil (an opioid) given during surgery and intraoperative EEG edge frequencies were significant predictors that linear models often overlook.
  • Dementia as a Red Flag: Across all timelines, a prior diagnosis of dementia or cognitive impairment remained the strongest categorical predictor of POD, highlighting the extreme vulnerability of this patient group.

Key Results

  • Superior Accuracy: The best-performing model (Tree-based M123) achieved a mean AUROC of 0.854 and a mean AUPRC of 0.418.
  • Baseline Comparison: These models significantly outperformed retrained versions of popular clinical baselines (like the Wassenaar and Boogaard models), which only achieved AUROCs around 0.61 in this specific recovery room context.
  • Prevalence: The study recorded a POD prevalence of 8.2% in the recovery room.
  • Top Predictors: The most discriminative numerical factors were the postoperative Aldrete score (a measure of recovery from anesthesia), patient age, and anesthesia duration.

Limitations and Future Work

The authors are transparent about the hurdles remaining for clinical adoption:

  • The Precision Gap: While the AUROC (ability to distinguish between classes) is high, the Precision remains relatively low (around 0.28 for the best model). In a real-world clinic, this means for every 10 patients the model flags, only about 3 might actually have POD. This "false alarm" rate is a major impediment to clinical implementation.
  • Single-Center Study: The data comes from one hospital system. While the sample size is large, the models need "external validation"—testing on data from different hospitals—to ensure they aren't overfitted to Charité’s specific practices.
  • Feature Availability: Some highly predictive features had high "missingness" rates. Future work could focus on streamlining which data points are mandatory in EHRs to improve model reliability.

The researchers have made their code and models openly accessible, inviting the global medical AI community to refine and implement these tools.

Summary updated

Abstract

Abstract Background Postoperative delirium (POD) contributes to severe outcomes such as death or development of dementia. Thus, it is desirable to identify vulnerable patients in advance during the perioperative phase. Previous studies mainly investigated risk factors for delirium during hospitalization and further used a linear logistic regression (LR) approach with time-invariant data. Studies have not investigated patients’ fluctuating conditions to support POD precautions. Objective In this single-center study, we aimed to predict POD in a recovery room setting with a non-linear machine learning (ML) technique using pre-, intra-, and postoperative data. Methods The target variable POD was defined with the Nursing Screening Delirium Scale (Nu-DESC) ≥ 1. Feature selection was conducted based on robust univariate test statistics and L 1 regularization. Non-linear multi-layer perceptron (MLP) as well as tree-based models were trained and evaluated – with the receiver operating characteristics curve (AUROC), the area under precision recall curve (AUPRC), and additional metrics – against LR and published models on bootstrapped testing data. Results The prevalence of POD was 8.2% in a sample of 73,181 surgeries performed between 2017 and 2020. Significant univariate impact factors were the preoperative ASA status, the intraoperative amount of given remifentanil, and the postoperative Aldrete score. The best model used pre-, intra-, and postoperative data. The tree-based model achieved a mean AUROC of 0.854 and a mean AUPRC of 0.418 outperforming linear LR, well as best applied and retrained baseline models. Conclusions Overall, non-linear machine learning models using data from multiple perioperative time phases were superior to traditional ones in predicting POD in the recovery room. Class imbalance was seen as a main impediment for model application in clinical practice. Author Summary Currently, the pathophysiology of postoperative delirium (POD) is unknown. Hence, there is no dedicated medication for treatment. Patients who experience POD are oftentimes mentally disturbed causing pressure on related family members, clinicians, and the health system. With our study, we want to detect POD before onset trying to give decision support to health professionals. Vulnerable patients could be transferred to delirium wards mitigating the risk of severe outcomes such as permanent cognitive decline. We also provide insides into clinical parameters - recorded before, during, and after the surgery - that could be adapted for reducing POD risk. Our work is openly available, developed for clinical implementation, and could be transferred to other clinical institutions.
BibTeX
@misc{giesa2023predicting,
  author = {Niklas Giesa and Stefan Haufe and Mario Menk and Björn Weiß and Claudia Spies and Sophie K. Piper and Felix Balzer and Sebastian Daniel Boie},
  title = {{Predicting postoperative delirium assessed by the Nursing Screening Delirium Scale in the recovery room for non-cardiac surgeries without craniotomy: A retrospective study using a machine learning approach}},
  year = {2023},
  doi = {10.1101/2023.11.21.23298802},
  howpublished = {medRxiv},
}
Figure