Robust Event Detection based on Spatio-Temporal Latent Action Unit using Skeletal Information

Hao Xing, Yuxuan Xue, Mingchuan Zhou, Darius Burschka

Introduction

Detecting human falls is a critical task for healthcare robotics and elderly monitoring systems. However, technical implementation faces two primary hurdles: sensor noise and action ambiguity. RGB-D cameras, such as Kinect or RealSense, often produce noisy 3D skeletal data due to depth measurement errors or occlusions. Furthermore, a fall can be visually similar to routine actions like "sitting down" or "lying down" when viewed as isolated poses. This paper proposes a robust framework using Gradual Online Dictionary Learning (GODL) to model the spatio-temporal structure of falls while effectively filtering out sensor noise.

System Architecture

The proposed system processes raw data through a multi-stage pipeline to extract meaningful action representations.

Overview of the proposed fall detection framework, from RGB-D input to the final fall-down models.

As shown in the architecture above, the process begins with extracting 2D skeletons via OpenPose, which are then mapped to 3D using registered depth images. After normalization to ensure invariance to the subject's position, the sequence is decomposed into five "latent action atoms": standing, bending knee, opening arm, knee landing, and arm supporting. These atoms represent the fundamental building blocks of a fall event.

Gradual Online Dictionary Learning (GODL)

The core innovation of this work is the GODL algorithm, which addresses the sensitivity of standard dictionary learning to outliers. By integrating Graduated Non-convexity (GNC) with a Geman-McClure (GM) cost function, the system assigns a weight to each skeletal frame during training.

Visualization of the weight decay process in GODL, where outlier skeletons (e.g., skeletons 5 and 6) are gradually assigned lower weights over iterations.

As illustrated in the figure above, the algorithm identifies frames that do not fit the learned pattern (outliers) and gradually reduces their influence. This ensures that the final dictionary of actions is derived from clean, representative data rather than sensor artifacts. To further reduce false positives, the system incorporates a temporal height model, requiring a sharp height change (the final height must be less than 50% of the initial height) to confirm a fall.

Experimental Results and Insights

The researchers evaluated GODL using the NTU RGB+D dataset, comparing it against several state-of-the-art methods including ODL, DLSI, and LRSDL.

  • Superior Performance: GODL achieved a state-of-the-art accuracy of 99.00% ± 0.36% and a precision of 95.62% ± 1.53%.
  • Noise Robustness: In tests with deliberate noise injection (up to 10%), GODL maintained stable performance and the lowest variance, whereas competing methods saw significant degradation.
  • Complexity-Aware Modeling: The study found that different action units require different dictionary dimensions; simple actions like "standing" need fewer parameters, while complex landing phases require more to avoid overfitting.
  • Explainability: Unlike "black-box" deep learning models, this approach allows for the tracking of specific sub-actions, providing transparency into the detection process.

Conclusion

The GODL framework provides a robust solution for fall detection by combining spatio-temporal action atoms with a noise-resilient learning algorithm. While currently focused on falls, the "action atom" approach holds potential for broader activity recognition. Future work will focus on automating threshold discovery and validating the system on larger, more diverse datasets to ensure generalizability across various environments.

Summary updated

BibTeX
@inproceedings{xing2021robust,
  author = {Hao Xing and Yuxuan Xue and Mingchuan Zhou and Darius Burschka},
  title = {{Robust Event Detection based on Spatio-Temporal Latent Action Unit using Skeletal Information}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636553},
}
Figure