Motion-Aware Heatmap Regression for Human Pose Estimation in Videos

Inpyo Song, Jongmin Lee, Moonwook Ryu, Jangwon LeeIJCAI 2024

Introduction

Human Pose Estimation (HPE) has seen significant advancements in static image processing, yet extending these capabilities to video remains a formidable challenge. In dynamic environments, such as sports or surveillance, fast-moving subjects often suffer from motion blur and frequent occlusions. Traditional models, which rely on static 2D Gaussian heatmaps to represent joint locations, often fail to capture the temporal nuances of movement. This paper introduces MTPose, a novel framework that leverages motion-aware heatmap regression to significantly improve pose estimation accuracy in video sequences.

The Limitation of Static Heatmaps

The standard approach in HPE is to represent the probability of a joint's location using a circular 2D Gaussian heatmap. This assumes that the uncertainty of a joint's position is uniform in all directions. However, in video, a fast-moving joint—like a runner's wrist—exhibits higher uncertainty along its trajectory than perpendicular to it. By ignoring these temporal motion cues, traditional models struggle to maintain precision during high-velocity actions.

MTPose: Adapting to Motion

The core innovation of MTPose is the transition from static circular heatmaps to dynamic, Motion-Aware Heatmaps. During training, the ground-truth heatmaps are adjusted based on motion vectors derived from adjacent frames.

For joints with minimal movement, the model maintains a tight circular heatmap. For rapidly moving joints, the heatmap transforms into an ellipse, stretched along the axis of motion (increasing standard deviation $\sigma_x$) and narrowed perpendicular to it (decreasing $\sigma_y$). This elliptical representation is then rotated to align with the joint's actual trajectory, effectively modeling the directional uncertainty caused by motion blur.

The MTPose framework generates elliptical heatmaps aligned with joint motion vectors to account for directional uncertainty.

Architecture

The MTPose architecture integrates spatial and temporal features through several key components:

  • Feature Extraction: A Vision Transformer (ViT) backbone extracts high-level spatial features, while an off-the-shelf optical flow model captures inter-frame movement.
  • Cross-Attention Fusion: A specialized module integrates spatial features with motion vectors to inform the heatmap shape.
  • Refinement: Deformable Convolutions (DCNv2) are employed to aggregate multi-scale information, producing the final refined pose estimation.

Experimental Results

MTPose was evaluated on major benchmarks, including PoseTrack2018, PoseTrack21, and Sub-JHMDB, setting new state-of-the-art (SOTA) records across the board. On the PoseTrack2018 dataset, the model achieved 89.0 mAP, representing a substantial +4.8 mAP improvement over previous methods.

An example of an elliptical heatmap accurately capturing the position of a fast-moving wrist in a video frame.

The method proved particularly effective for "high-dynamic" joints. For instance, accuracy for wrists and ankles on PoseTrack2018 increased by 7.1 mAP and 8.3 mAP, respectively. Despite the integration of optical flow and attention mechanisms, MTPose maintains an efficient processing speed of approximately 14 frames per second (FPS), making it suitable for near-real-time applications.

Conclusion

MTPose demonstrates that motion blur in video is not merely noise but a valuable directional signal. By redefining the target representation from static circles to motion-aware ellipses, the framework provides a simple yet powerful solution to the challenges of video-based human pose estimation. Future research may explore extending this directional uncertainty model to bottom-up pose estimation and 3D pose reconstruction from 2D video sequences.

Summary updated

BibTeX
@inproceedings{song2024motion,
  author = {Inpyo Song and Jongmin Lee and Moonwook Ryu and Jangwon Lee},
  title = {{Motion-Aware Heatmap Regression for Human Pose Estimation in Videos}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure