SONAR, a nursing activity dataset with inertial sensors

Orhan Konak, Valentin Döring, Tobias Fiedler, Lucas Liebe, Leander Masopust, Kirill Postnov, F. Sauerwald, Felix Treykorn, Alexander Wischmann, Stefan Kalabakov, Hristijan Gjoreski, Mitja Luštrek, Bert ArnrichScientific 2023 · DOI

TL;DR

This paper introduces SONAR, a novel dataset of nursing activities captured using inertial sensors in a real-world nursing home setting. It addresses the critical need for accurate nursing documentation and improved patient care by providing a rich dataset and benchmark models for human activity recognition in this complex domain.

What Problem Does This Paper Address?

Accurate and comprehensive documentation is vital for quality patient care, but it often contributes to a significant nursing workload. Human Activity Recognition (HAR) using wearable sensors offers a potential solution by automating the recording of nursing tasks. However, existing HAR datasets often focus on simpler activities of daily living (ADL) or are collected under controlled laboratory conditions, failing to capture the nuances of real-world nursing environments. This gap limits the development of robust HAR systems for healthcare. The SONAR dataset aims to bridge this gap by providing a comprehensive collection of nursing activities performed by professional nurses in an actual elderly care facility.

Methods

The researchers collected data using five Xsens DOT v2 inertial measurement units (IMUs) worn by 14 professional nurses. These sensors captured acceleration, angular velocity, and orientation data. To ensure accurate labeling, a custom application was developed that synchronized sensor data with real-time pose estimation from parallel video streams. This allowed for precise annotation of 23 distinct nursing activities. The study also involved training and evaluating three deep learning models: CNN-LSTM, ResNet, and DeepConvLSTM, to benchmark the dataset's utility for activity recognition. Various cross-validation techniques (k-fold, leave-recordings-out, leave-one-subject-out) and performance metrics (accuracy and F1-score) were employed to assess model performance across different sensor combinations.

Datasets

The primary dataset introduced in this paper is SONAR (Sensor-Oriented Nursing Activity Recognition). This dataset comprises 61.7 hours of data from 14 nurses performing 23 distinct nursing activities in a real-world nursing home. The data includes 14 sensor streams (acceleration and angular velocity) from five IMUs. The paper also references several other HAR datasets for comparison, including OPPORTUNITY, PAMAP2, Skoda, UCI HAR, SHL, and HARTH.

Key Insights and Ideas

  • Real-world complexity matters: The SONAR dataset highlights that activities in a real nursing home are far more complex and varied than those typically found in laboratory settings or datasets focused on ADLs. This complexity is crucial for developing truly applicable HAR systems.
  • Sensor fusion is key: The study demonstrates that combining data from multiple sensors, particularly wrist and pelvis sensors, significantly improves activity recognition performance. This suggests that a multi-sensor approach is essential for capturing the subtle movements involved in nursing tasks.
  • Deep learning models show promise: The benchmark results indicate that deep learning models, especially CNN-LSTM, can achieve reasonable performance in recognizing nursing activities from inertial sensor data, even with the inherent challenges of real-world data.
  • Pose estimation aids labeling: The innovative use of pose estimation for data labeling proved effective in ensuring accurate and privacy-preserving annotation, a critical aspect for sensitive healthcare data.

Key Results

The SONAR dataset contains 61.7 hours of data, encompassing 23 nursing activities performed by 14 nurses. The benchmark evaluation showed that the CNN-LSTM model achieved the best performance across various sensor combinations and cross-validation strategies. For instance, with a combination of wrist, pelvis, and ankle sensors ({LW, RW, ST, LF, RF}), the CNN-LSTM model achieved an accuracy of 0.44 (±0.06) and an F1-score of 0.4 (±0.06) using k-fold cross-validation. The study also found that wrist sensors (LW and RW) were particularly crucial for performance, with the addition of the pelvis sensor (ST) further enhancing results.

Limitations and Future Work

  • Dataset size: While 61.7 hours is substantial, the authors acknowledge that larger datasets would further improve model generalization.
  • Activity granularity: The current dataset categorizes activities into 23 distinct types. Future work could explore finer-grained activity recognition.
  • Generalizability: While collected in a real-world setting, the dataset represents one specific nursing home. Further validation across different facilities and patient populations would be beneficial.
  • Model complexity: The paper focuses on three deep learning models. Exploring other architectures, including transformer-based models, could yield further improvements.
  • Privacy considerations: While pose estimation aids anonymization, the ethical implications of continuous monitoring in healthcare settings warrant ongoing consideration.

The authors suggest that the SONAR dataset can be used to develop more sophisticated HAR systems, potentially leading to improved nursing documentation, reduced workload, and ultimately, enhanced patient care. The availability of the dataset and associated code on Zenodo and GitHub facilitates further research and collaboration in this important area.

Summary updated

Abstract

Accurate and comprehensive nursing documentation is essential to ensure quality patient care. To streamline this process, we present SONAR, a publicly available dataset of nursing activities recorded using inertial sensors in a nursing home. The dataset includes 14 sensor streams, such as acceleration and angular velocity, and 23 activities recorded by 14 caregivers using five sensors for 61.7 hours. The caregivers wore the sensors as they performed their daily tasks, allowing for continuous monitoring of their activities. We additionally provide machine learning models that recognize the nursing activities given the sensor data. In particular, we present benchmarks for three deep learning model architectures and evaluate their performance using different metrics and sensor locations. Our dataset, which can be used for research on sensor-based human activity recognition in real-world settings, has the potential to improve nursing care by providing valuable insights that can identify areas for improvement, facilitate accurate documentation, and tailor care to specific patient conditions.
BibTeX
@inproceedings{konak2023sonar,
  author = {Orhan Konak and Valentin Döring and Tobias Fiedler and Lucas Liebe and Leander Masopust and Kirill Postnov and F. Sauerwald and Felix Treykorn and Alexander Wischmann and Stefan Kalabakov and Hristijan Gjoreski and Mitja Luštrek and Bert Arnrich},
  title = {{SONAR, a nursing activity dataset with inertial sensors}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41597-023-02620-2},
}
Figure