Constraint-Based Optimized Human Skeleton Extraction from Single-Depth Camera

Ruotong Li, Weixin Si, Michael Weinmann, Reinhard Klein2019 · DOI

TL;DR

This paper introduces a novel approach to refine human skeleton extraction from single-depth cameras, addressing inaccuracies and inconsistencies often found in raw SDK outputs. By incorporating bone length constancy and skeletal symmetry constraints, along with temporal predictions, the method significantly improves the stability and accuracy of pose estimation, particularly in challenging scenarios with occlusions.

What Problem Does This Paper Address?

Extracting accurate and consistent human skeletons from single-depth camera data is a persistent challenge in computer vision. Existing methods, like those provided by SDKs such as Kinect or Nuitrack, often produce skeletons with fluctuating bone lengths and asymmetrical body parts, especially when dealing with occlusions, varying appearances, and sensor noise. This inconsistency hinders reliable human behavior analysis for applications like surveillance, human-robot interaction, and gaming. The paper aims to bridge this gap by developing a post-processing refinement technique that leverages inherent human skeletal properties to correct these errors.

Methods

The core of the proposed method is a constraint-based optimization framework that refines initial skeleton estimates obtained from existing SDKs. The pipeline involves several key steps:

  • Initial Per-Frame Pose Extraction: The process begins with raw skeleton data from SDKs like Kinect or Nuitrack.
  • Bone Length Initialization: The first few frames of the sequence are used to establish reference bone lengths, considering symmetry and temporal stability. Weights are assigned to bone length observations based on their orientation relative to the camera, assuming bones more perpendicular to the camera are more reliably captured.
  • Trust Data Detection: For subsequent frames, detected bone lengths are compared to the initialized references. If deviations are significant or joints are not reliably tracked by the SDK, those parts are flagged as unreliable.
  • Joint Position Prediction: To handle unreliable joints, the method employs a combination of forward and backward temporal predictions. This leverages the inherent smoothness of human motion to estimate likely joint positions, providing more robust predictions than simple interpolation.
  • Pose Refinement: A nonlinear optimization framework is used to refine the skeleton. This framework minimizes an energy function that penalizes deviations from the initial joint estimates and predicted positions, while simultaneously enforcing the established bone length and symmetry constraints. A quasi-Newton solver is employed for this minimization.

Datasets

The paper primarily uses RGB-D video inputs provided by a Kinect v2.0. The initial skeleton and point cloud data are sourced from the Kinect SDK [1]. The authors also mention the potential applicability to data from Nuitrack SDK for Intel RealSense D435. No new datasets were introduced; the focus is on refining existing data.

Key Insights and Ideas

  • Leveraging Inherent Constraints: The paper highlights that human skeletons possess inherent properties like constant bone lengths and left-right symmetry, which are often ignored by initial pose estimation algorithms. Exploiting these constraints is crucial for robust refinement.
  • Confidence-Based Weighting: The idea of weighting bone length observations based on their orientation relative to the camera is insightful. It acknowledges that depth camera accuracy can vary with viewing angle, leading to more reliable reference bone length estimations.
  • Combined Forward-Backward Prediction: Using both forward and backward temporal predictions for joint positions offers a more robust way to handle occlusions and sensor noise compared to unidirectional prediction, even at the cost of a slight temporal delay.
  • Energy Minimization for Consistency: Formulating the refinement as an energy minimization problem allows for a principled way to balance multiple constraints (initial estimates, temporal predictions, skeletal properties) and find an optimal, consistent skeleton.

Key Results

  • The proposed method significantly reduces the variance of bone lengths in the estimated skeletons compared to raw SDK outputs. Figures 11-15 show substantial reductions in bone length variance for various bone sections across different datasets.
  • The refined skeletons are observed to be more stable and accurate, as demonstrated visually in Figures 17 and 18, especially when compared against the point cloud data.
  • The method effectively detects and compensates for erroneous joint location estimates, particularly those caused by occlusions or tracking failures, by using temporal predictions.
  • The approach is device-independent and can be integrated with different skeleton extraction SDKs.

Limitations and Future Work

  • Limited Error Recovery: The current technique can only recover from up to four erroneous frames in a pose sequence. This limitation is significant for self-collision motions like turning around, which can span more than four frames.
  • Failure Cases: The paper acknowledges failure cases, such as when the actor performs a turn-around movement, which can lead to a left-right flip in the skeleton representation (Figure 19).
  • Temporal Delay: The use of second-order approximations for temporal prediction introduces a delay of three frames (approximately 0.1 seconds at 30 fps).
  • Future Work: The authors plan to address the limitations by coupling convolutional neural networks (CNNs) with long short-term memory networks (LSTMs) for continuous pose estimation refinement. They also aim to refine the handling of partial flips in motion skeletons.

Summary updated

BibTeX
@misc{li2019constraint,
  author = {Ruotong Li and Weixin Si and Michael Weinmann and Reinhard Klein},
  title = {{Constraint-Based Optimized Human Skeleton Extraction from Single-Depth Camera}},
  year = {2019},
  doi = {10.3390/s19112604},
}
Figure