Abstract
We present a method for human pose tracking that is based on learning spatiotemporal relationships among joints. Beyond generating the heatmap of a joint in a given frame, our system also learns to predict the offset of the joint from a neighboring joint in the frame. Additionally, it is trained to predict the displacement of the joint from its position in the previous frame, in a manner that can account for possibly changing joint appearance, unlike optical flow. These relational cues in the spatial domain and temporal domain are inferred in a robust manner by attending only to relevant areas in the video frames. By explicitly learning and exploiting these joint relationships, our system achieves state-of-the-art performance on standard benchmarks for various pose tracking tasks including 3D body pose tracking in RGB video, 3D hand pose tracking in depth sequences, and 3D hand gesture tracking in RGB video.
Introduction
Tracking human pose in video sequences is significantly more challenging than single-image pose estimation. Standard per-frame detectors often struggle with motion blur, self-occlusion, and uncommon body configurations. Historically, researchers have relied on temporal smoothness constraints or generic optical flow to bridge the gap between frames. However, these methods are often "blind" to the semantic structure of the human body. Optical flow, for instance, frequently fails when a limb rotates or its appearance changes significantly.
This paper introduces a novel framework that explicitly learns the spatiotemporal relationships between joints. By training a neural network to predict both "bone vectors" (spatial offsets between joints) and "joint displacements" (temporal movement between frames), the system moves beyond simple heatmap estimation to a more robust, semantically aware tracking mechanism.
Methodology: Explicit Spatiotemporal Joint Relation Learning
The core innovation is a network designed to predict how joints relate to one another in space and how they deform over time. Instead of just asking "where is the hand?", the model learns "how far is the hand from the elbow?" (spatial) and "where did the hand move since the last frame?" (temporal).
Key Technical Innovations
- Per-pixel Relation Maps: The network generates dense maps where every pixel predicts a relative offset. This provides a rich set of candidates for joint locations.
- Attention-like Pixel Weighting: Not all pixels are equally informative. The authors implement a weighting mechanism ($W$) that focuses the model on pixels near the "related" joint. For example, the area around the elbow is the most informative for predicting the wrist's position. This "JointOne" strategy effectively filters out background noise.
- Unified Optimization: During inference, the system combines three orthogonal cues—single-frame heatmaps, spatial bone vectors, and temporal displacements—into a linear least-squares problem. This redundancy allows the tracker to maintain accuracy even when individual cues (like heatmaps) become noisy due to blur or occlusion.
Experimental Results
The method was evaluated on four diverse benchmarks: Human3.6M (3D body), MSRA Hand 2015 (depth-based hand), Dexter+Object (hand-object interaction), and Stereo Hand Pose (RGB-D hand gestures).
Performance Gains
- Human3.6M: The system improved the state-of-the-art (SOTA) by 4.1mm (8.3% relative) with extra 2D data, and by 6.0mm (9.4% relative) using only 3D data.
- MSRA15: The mean 3D joint error was reduced from 7.23mm to 5.96mm, representing a massive 17.6% improvement.
- Ablation Studies: As shown in the figures above, the per-pixel regression with attention (JointOne) outperformed standard fully connected baselines by approximately 18-22% in both spatial and temporal tasks.
Key Insights
- Semantic Motion vs. Generic Flow: Learning joint-specific displacements is superior to generic optical flow because the network accounts for semantic appearance changes (e.g., a hand clenching into a fist).
- Sparse Attention: Forcing the network to attend to relevant local patches (the "JointOne" strategy) is more effective than holistic global mapping, which can introduce irrelevant features.
- Multi-Step Temporal Cues: The system's ability to look several frames back and forward increases robustness by providing more observations for the final pose estimate.
Conclusion and Future Work
While achieving state-of-the-art results, the framework has areas for further development. The final optimization step involves solving a linear system across multiple frames, which may pose challenges for real-time performance on low-power devices. Additionally, the current model is optimized for single-subject tracking; extending it to multi-person scenarios with overlapping joints would require additional association logic. Finally, as the relation learning acts as a refinement step, it remains somewhat dependent on the quality of initial detections.
Summary updated