Abstract
We present a method for human pose tracking that is based on learning spatiotemporal relationships among joints. Beyond generating the heatmap of a joint in a given frame, our system also learns to predict the offset of the joint from a neighboring joint in the frame. Additionally, it is trained to predict the displacement of the joint from its position in the previous frame, in a manner that can account for possibly changing joint appearance, unlike optical flow. These relational cues in the spatial domain and temporal domain are inferred in a robust manner by attending only to relevant areas in the video frames. By explicitly learning and exploiting these joint relationships, our system achieves state-of-the-art performance on standard benchmarks for various pose tracking tasks including 3D body pose tracking in RGB video, 3D hand pose tracking in depth sequences, and 3D hand gesture tracking in RGB video.
Introduction
Human pose tracking in video sequences is significantly more challenging than static pose estimation due to motion blur, self-occlusion, and rapid movements. Traditional methods often rely on temporal smoothness or generic optical flow, but these lack anatomical awareness. Optical flow tracks pixels rather than joints and frequently fails when a joint's appearance changes due to rotation or lighting. This paper addresses these gaps by teaching neural networks to explicitly predict spatiotemporal relationships: how joints relate to one another spatially (bone vectors) and how they move through time (displacement).
Methodology
The proposed framework predicts three orthogonal cues for every joint: single-frame heatmaps for location, spatial relations (bone vectors from parent to child joints), and temporal relations (joint displacement from frame $t-d$ to $t$).
Per-pixel Regression and Attention
To improve robustness against noise, the system utilizes per-pixel regression. Instead of outputting a single coordinate, the network generates "Relation Maps" where every pixel predicts the joint's offset. A critical component is the Pixel Weighting Mechanism, which acts as an attention module. It assigns higher weights to pixels near the joint and filters out distant, irrelevant background noise.
The final 3D trajectory is produced via a global optimization step that balances heatmap predictions with these learned spatial and temporal constraints using a linear least squares solver.
Experimental Results
The model was evaluated on major benchmarks including Human3.6M (body pose) and MSRA Hand 2015 (depth-based hand pose). The system achieved state-of-the-art performance, reducing the Mean Per Joint Position Error (MPJPE) on Human3.6M to 45.5mm and improving MSRA Hand results by 17.6% relative to previous bests.
Key Insights
A significant finding is that joint-specific motion tracking is far more robust than generic optical flow. Ablation studies revealed that the "Ours-JointOne" strategy—which focuses on pixels specifically at joint locations—consistently outperformed holistic baselines. This suggests that local context is a more reliable predictor for pose than global image features, which often act as distractors.
Conclusion
By explicitly modeling bone vectors and joint displacements with an attention-weighted regression scheme, this framework provides a robust solution for 3D pose tracking. Future work includes extending the framework to multi-person scenarios and exploring non-linear optimization to capture even tighter anatomical constraints.
Summary updated