Towards Robust and Accurate Single-View Fast Human Motion Capture

Tao Yu, Jianhui Zhao, Yuanhao Huang, Yipeng Li, Yebin LiuIEEE Access 2019 · DOI

Introduction

Capturing high-fidelity human motion typically requires expensive multi-camera setups and controlled studio environments. This paper presents a robust system designed to achieve accurate human motion capture (MoCap) using only a single RGB-D sensor, such as a Microsoft Kinect. By integrating real-time tracking with deep-learning-based pose detection and a novel bidirectional post-processing technique, the system overcomes the traditional limitations of single-view setups, particularly during fast movements and complex self-occlusions.

The Challenge of Single-View Capture

Single-view solutions are highly accessible but face two primary technical hurdles:

  1. Fast Motion: Rapid movements often result in image blur and significant data loss in depth sensors. Sequential tracking algorithms frequently "lose" the subject under these conditions and struggle to recover.
  2. Self-Occlusion: When a limb moves behind the torso, it disappears from the camera's view. Traditional trackers often fail to correctly re-acquire the limb when it suddenly reappears.
Depth map input from an RGB-D sensor

While learning-based methods can estimate poses frame-by-frame, they often produce jittery results. Conversely, traditional tracking methods provide smoothness but are fragile. This research aims to bridge the gap between these two approaches.

Proposed Methodology

The authors introduce a two-stage pipeline to ensure both temporal smoothness and robustness to failure.

Semantic Hybrid Motion Capture (Forward Tracking)

The system employs a "Semantic Hybrid" approach that combines tracking and detection. An Inspector module monitors the "semantic tracking loss" for four specific limb groups (arms and legs). If the tracker begins to fail for a specific limb, the system utilizes a deep-learning pose detector (HMR) to reinitialize only that failing part. This limb-level reinitialization is significantly more stable than a global skeleton reset, which often introduces noise into parts of the body that were already well-tracked.

Semantic Bidirectional Motion Blending

To address the difficulty of tracking limbs that suddenly reappear, the authors propose a "time-reversal" strategy. They observed that while tracking a limb's reappearance is difficult, tracking its gradual disappearance is much more reliable. By running the tracking algorithm both forward and backward through the video sequence, the system can leverage "future" information to resolve current ambiguities. The forward and backward results are then blended based on their respective confidence scores to produce a smooth, accurate final motion.

Results and Evaluation

The system was evaluated using custom fast-motion sequences (e.g., basketball, kicking) and the BodyFusionVicon dataset, which provides ground-truth data from a professional VICON system.

Final skeleton tracking output on a human subject

Key results include:

  • Improved Accuracy: The method demonstrated significantly more stable error curves compared to state-of-the-art solutions like DoubleFusion.
  • Robustness: In fast-motion tests, bidirectional blending reduced average tracking loss by over 50% compared to standard tracking-only methods.
  • Visual Quality: The system successfully tracked complex maneuvers, such as handstands and rapid dribbling, where previous methods often resulted in "tangled" or detached limb artifacts.

Conclusion

This research provides a significant step toward making high-quality motion capture more accessible. By combining local limb-level reinitialization with bidirectional temporal blending, the system achieves robustness previously unseen in single-view setups. While challenges remain regarding loose clothing and extreme occlusions, the approach demonstrates the power of combining semantic body knowledge with temporal consistency.

Summary updated

BibTeX
@article{yu2019towards,
  author = {Tao Yu and Jianhui Zhao and Yuanhao Huang and Yipeng Li and Yebin Liu},
  title = {{Towards Robust and Accurate Single-View Fast Human Motion Capture}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2920633},
}
Figure