Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video

Runyang Feng, Haoming Chen

Abstract

Video-based human pose estimation (VPE) faces significant challenges from motion blur, occlusions, and rapid movements. Traditional methods often rely on pixel-level matching, such as optical flow, which fails when visual data is degraded. This paper introduces the SpatioTemporal Collaboration (SDTC) framework, which shifts the focus from pixel-level tracking to high-level semantic understanding. By employing a masked-reconstruction strategy to learn motion patterns and a mutual learning module to fuse appearance and motion features, SDTC achieves state-of-the-art results on the PoseTrack benchmarks while maintaining computational efficiency.

Introduction

Human pose estimation in video is inherently more complex than in static images. While a single frame might provide a clear view of a subject, subsequent frames often suffer from motion blur or environmental occlusions. The prevailing paradigm in VPE has been to track specific pixels or color blobs across frames. However, this approach is fundamentally brittle; if a limb is hidden or blurred beyond recognition, there are no reliable pixels to match.

To address these limitations, we propose that models must understand the "semantical dynamics" of the human body—the underlying logic and constraints of how humans move. Instead of asking "where did this pixel go?", the model should ask "given the previous motion, where should this joint logically be?" The SDTC framework implements this shift through two primary innovations: the Multi-Level Semantic Motion Encoder (MLSME) and the Spatial-Motion Mutual Learning (SMML) module.

Methodology

Multi-Level Semantic Motion Encoder (MLSME)

Inspired by the success of masked signal modeling in natural language processing (BERT) and computer vision (MAE), the MLSME uses a self-supervised strategy to learn robust motion representations. During training, the model "hides" random 3D cubes of video patches and entire frames. To successfully reconstruct these missing segments, the model is forced to learn the global and local correlations of human movement.

This hierarchical approach allows the model to capture both fine-grained patch-level details and broader frame-level dynamics. By learning the "grammar" of human motion, the encoder becomes resilient to temporary occlusions and blur, as it can infer missing information based on the learned semantic context.

Spatial-Motion Mutual Learning (SMML)

Effective VPE requires a sophisticated fusion of spatial features (what a person looks like) and motion features (how they are moving). Rather than simple concatenation, the SMML module treats these modalities as complementary partners through three stages:

  1. Self-Feature Refinement: Enhances the internal representation of each modality.
  2. Cross-Feature Propagation: Utilizes a Spatial-Motion Cross-Attention (SMCA) mechanism to allow the modalities to exchange context. This helps the motion features inform the spatial features during blur, and vice versa during static occlusions.
  3. Adaptive Feature Fusion: A gating mechanism determines the optimal weight for each modality at a pixel-level. In clear frames, appearance features may dominate; in high-speed sequences, the model shifts its reliance toward motion context.

Experimental Results

The SDTC framework was evaluated on the PoseTrack2017, PoseTrack2018, and PoseTrack21 datasets, which represent the most challenging benchmarks for video pose estimation and tracking.

Performance Benchmarks

SDTC established new state-of-the-art (SOTA) records across all three datasets:

  • PoseTrack2017: Achieved 87.5 mAP, outperforming the previous leading method (TDMI) by 1.8 points.
  • PoseTrack2018: Reached 84.3 mAP, showing a significant improvement over established models like DSTA and FAMI-Pose.
  • PoseTrack21: Achieved 84.9 mAP, demonstrating robustness in complex, crowded scenes and with small-scale subjects.

Efficiency

Beyond accuracy, SDTC is designed for practical deployment. The model utilizes 10.4M parameters and requires 177 GFLOPs. This makes it significantly lighter and more computationally efficient than several previous SOTA models, such as SLT-Pose, without sacrificing performance.

Discussion and Conclusion

The success of SDTC highlights the importance of moving beyond local pixel similarity toward global semantic correlation. By treating motion as a structured signal to be reconstructed rather than just a difference between frames, the model gains a deeper understanding of human kinetics.

While the framework is highly robust, it faces limitations when dealing with extremely incomplete human bodies (e.g., extreme close-ups where only a few joints are visible), as the semantic logic of the pose can become ambiguous. Future research will explore extending this semantical dynamics approach to 3D human pose estimation and multi-person tracking, where temporal consistency is paramount. In conclusion, SDTC provides a powerful, efficient, and semantically-aware foundation for the next generation of video-based human pose analysis.

Summary updated

BibTeX
@article{feng2025learning,
  author = {Runyang Feng and Haoming Chen},
  title = {{Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video}},
  journal = {Neurocomputing},
  year = {2025},
  doi = {10.1016/j.neucom.2025.129581},
}
Figure