A computer vision system for deep learning-based detection of patient mobilization activities in the ICU

Serena Yeung, Francesca Rinaldo, Jeffrey Jopling, Bingbin Liu, Rishab Mehra, N. Lance Downing, Michelle Guo, Gabriel M. Bianconi, Alexandre Alahi, Julia Lee, Brandi Campbell, Kayla Deru, William Beninati, Li Fei-Fei, Arnold MilsteinNpj Digit. Medicine 2019 · DOI

Introduction

Early mobilization is critical for patients in the Intensive Care Unit (ICU) to prevent "post-intensive care syndrome"—a cluster of cognitive and physical impairments that often follow prolonged hospital stays. Despite its importance, clinicians currently lack efficient, scalable tools to track these movements. Traditional methods rely on manual observation or retrospective Electronic Health Record (EHR) mining, both of which are labor-intensive, prone to documentation lags, and often lack the granularity needed to assess the intensity of mobilization, such as duration or the number of assisting staff.

To address this, researchers developed a computer vision system using privacy-preserving depth sensors and deep learning to automatically track patient mobilization. This system provides a continuous, objective audit of activity, offering a scalable alternative to manual logging.

Methods

The Computer Vision Technology (CVT) pipeline consists of two primary components: activity detection and personnel quantification.

Activity Detection and Duration

The system utilizes a multi-label recurrent convolutional neural network. Spatial features are extracted from individual frames using an 18-layer ResNet pre-trained on ImageNet. These features are then processed by a two-layer bidirectional Long Short-Term Memory (LSTM) network to capture temporal sequences, allowing the model to distinguish between similar actions like sitting down versus standing up. An ensemble of six models produces the final activity classification.

Healthcare Personnel Quantification

To estimate the number of staff required for assistance, the system employs the YOLOv2 (You Only Look Once) object detection architecture. The model detects the spatial locations of people in the room; the maximum count during a mobility event (excluding the patient) determines the staff count.

Privacy and Data

A key feature of the system is the use of depth sensors instead of standard RGB cameras. These sensors capture 3D silhouettes, obscuring identifiable facial features and ensuring the data is "privacy-safe" for 24/7 clinical monitoring.

Sequence of depth sensor frames illustrating a mobility event while preserving patient privacy.

The primary dataset was collected at the Intermountain LDS Hospital ICU, comprising 563 real-world mobility instances. This was augmented with 318 simulated events from a dedicated facility at Stanford University to address the "small data" challenge common in medical AI.

Results

The system demonstrated high accuracy in detecting mobilization activities and estimating their duration.

Activity Detection Performance

The model achieved a mean sensitivity of 87.2% and a specificity of 89.2% across four key activities: getting in/out of bed and getting in/out of a chair. The overall Area Under the Curve (AUC) was 0.938, indicating robust performance across all monitored activities.

ROC curves and AUC values for the four detected mobilization activities.

Duration and Personnel Counting

The algorithm predicted a mean duration of 7.6 seconds, closely aligning with the ground-truth manual annotation of 9.0 seconds. Personnel counting achieved a mean accuracy of 68.8%. While the system was highly accurate when a patient moved alone (75%) or with one staff member (74%), performance decreased during more complex events where multiple staff members overlapped in the sensor's field of view, causing occlusion.

Discussion

The use of depth sensors represents a significant ethical and technical advantage, allowing for continuous monitoring without violating HIPAA or patient privacy. Furthermore, by capturing the duration and staff count, the system provides a measure of mobilization "intensity," which is a stronger predictor of clinical outcomes than binary movement flags.

However, challenges remain, particularly regarding occlusion during personnel counting. Future work may involve multi-sensor fusion to provide multiple viewpoints within a single room. Additionally, while the current model performed well, fine-tuning may be necessary for different ICU layouts. The ultimate goal is to integrate this real-time data into clinical workflows, alerting care teams when mobilization protocols are not being met.

Summary updated

BibTeX
@article{yeung2019computer,
  author = {Serena Yeung and Francesca Rinaldo and Jeffrey Jopling and Bingbin Liu and Rishab Mehra and N. Lance Downing and Michelle Guo and Gabriel M. Bianconi and Alexandre Alahi and Julia Lee and Brandi Campbell and Kayla Deru and William Beninati and Li Fei-Fei and Arnold Milstein},
  title = {{A computer vision system for deep learning-based detection of patient mobilization activities in the ICU}},
  journal = {Npj Digit. Medicine},
  year = {2019},
  doi = {10.1038/s41746-019-0087-z},
}
Figure