TL;DR
This paper introduces a novel method for automatically detecting clinical interventions in the Neonatal Intensive Care Unit (NICU) using only depth camera data and Vision Transformers (ViTs). This non-contact approach enhances patient privacy and robustness to lighting changes, paving the way for more efficient patient monitoring and automated clinical charting.
What Problem Does This Paper Address?
The Neonatal Intensive Care Unit (NICU) requires continuous monitoring of vulnerable infants. Traditional methods often rely on wired sensors that can be intrusive, cause skin irritation, and require frequent removal during necessary interventions. While video-based monitoring offers a non-contact alternative, existing methods often use RGB or RGB-D data, which can raise privacy concerns and are sensitive to lighting variations. Furthermore, periods of clinical intervention disrupt continuous monitoring and are often excluded from analysis, hindering the development of robust monitoring systems. This paper aims to address these limitations by developing a privacy-preserving, lighting-robust method for detecting these critical intervention periods using only depth data.
Methods
The core of this research is the application of Vision Transformers (ViTs) for intervention detection. ViTs, known for their success in image classification, were adapted to process depth data. The authors explored several design parameters to optimize performance:
- Depth Data Encoding: Investigated different ways to represent depth data for the ViT. Notably, they explored HHA (Horizontal disparity, Height above ground, and Angle with gravity) encoding, which transforms the single-channel depth image into a three-channel representation, inspired by its success in RGB-D image processing.
- Perspective Transformation (PT): Implemented a PT algorithm to correct for non-optimal camera placement, which can be common in NICU environments. This aims to standardize the input data regardless of camera angle.
- Simulated Data: Introduced simulated data from a neonatal manikin to address the class imbalance in the real-world NICU dataset, where "no intervention" frames significantly outnumbered "intervention" frames.
- Model Architectures: Utilized two sizes of pre-trained ViT models: 'vit_tiny' (~5.4M parameters) and 'vit_base' (~85M parameters).
The study employed a rigorous 5-fold cross-validation approach, repeated five times, to ensure robust evaluation of model performance across different design parameter combinations.
Datasets
The study utilized two datasets:
- NICU Data: Real-world clinical data collected from 27 neonatal patients at the Children's Hospital of Eastern Ontario (CHEO). This dataset comprises 14,892 images, with a significant class imbalance (10.8:1) favoring "no intervention" frames. Due to ethical restrictions, this dataset cannot be publicly released.
- Simulated Data: 600 depth images generated using a neonatal manikin and a depth camera. This data was used to augment the "intervention" class in the NICU dataset, reducing the class imbalance to approximately 7.3:1.
Key Insights and Ideas
- ViTs for Depth-Based Intervention Detection: The paper demonstrates that ViTs are a viable and effective architecture for intervention detection using depth data, outperforming previous CNN-based approaches in several metrics.
- HHA Encoding's Nuanced Impact: While HHA encoding improved performance for the ViT models, its effect on the baseline CNN model was mixed, suggesting that architecture plays a crucial role in how effectively this encoding is utilized.
- Perspective Transformation's Limited Benefit for ViTs: Unlike CNNs, where PT showed improvements, its impact on ViT performance was not consistently significant, possibly due to the patch-based processing of ViTs.
- Simulated Data's Unexpected Outcome: Supplementing the training data with simulated images did not lead to an overall performance improvement for the ViT models, contrary to expectations. This highlights the challenges of domain adaptation and the potential for subtle differences between simulated and real-world data.
- Model Size Matters: The larger 'vit_base' model consistently outperformed the 'vit_tiny' model across most metrics, indicating that a greater number of trainable parameters is beneficial for this task.
Key Results
The best-performing model, a 'vit_base' model leveraging both perspective transform and HHA encoding, achieved:
- Sensitivity: 85.6%
- Precision: 89.8%
- F1-Score: 87.6%
This model significantly outperformed baseline CNN models, including a multi-modal RGB-D fusion model and an exclusively depth-based CNN. The 'vit_base' model with HHA encoding and perspective transformation achieved the highest overall performance.
Limitations and Future Work
Limitations:
- Simulated Data Performance: The addition of simulated data did not improve ViT performance, suggesting potential domain differences or an insufficient simulation strategy.
- Perspective Transformation for ViTs: The benefits of perspective transformation were not as pronounced for ViTs as they were for CNNs, indicating that ViT architectures may not leverage this preprocessing step in the same way.
- Class Imbalance: While addressed with simulated data, the inherent class imbalance remains a challenge.
Future Work:
- More Diverse Simulated Data: Explore more diverse simulated data from various care settings and environments comparable to real-world NICU conditions.
- Hyperparameter Tuning for HHA: Expand hyperparameter search space for HHA encoding to potentially unlock its full benefits for ViTs.
- Foundation Models for HHA: Investigate training or identifying foundation models pre-trained on HHA-encoded data.
- Patch Size Optimization: Study the effect of varying patch sizes in ViTs for improved performance.
- Background Subtraction: Explore background subtraction techniques to better highlight depth differences during interventions.
- Video Data Analysis: Extend the work to depth video, leveraging recent advancements in ViTs for video analysis, to capture temporal dynamics of interventions.
- Precise Intervention Start/End Detection: Develop methods for more precise determination of intervention start and end times, which is challenging even for human annotators.
Summary updated