FastPose: Towards Real-time Pose Estimation and Tracking via Scale-normalized Multi-task Networks

Jiabin Zhang, Zheng Zhu, Wei Zou, Peng Li, Yanwei Li, Hu Su, Guan HuangarXiv 2019 · arXiv

Introduction

Multi-person pose tracking is a critical task in computer vision, yet achieving high accuracy in real-time remains challenging. Traditional "top-down" methods are accurate but slow due to their multi-stage nature. Conversely, unified frameworks are faster but often struggle with scale variation. FastPose bridges this gap by integrating human detection, pose estimation, and person re-identification (Re-ID) into a single, efficient multi-task network.

Methods: Multi-tasking and Scale Normalization

The primary hurdles in unified pose tracking are scale variation—where people appear at different sizes—and occlusions. FastPose introduces three key innovations to solve these:

  1. Multi-task Network (MTN): A single backbone (e.g., ResNet or MobileNet) supports three specialized heads for detection, 2D pose estimation, and Re-ID feature extraction. This shared architecture significantly reduces the marginal cost of adding tracking and pose estimation to a detector.
  2. Scale-normalized Image and Feature Pyramid (SIFP): This strategy ensures the network learns to process humans at consistent, optimal resolutions by ignoring extreme scales during specific training phases. During inference, it handles different sizes without the overhead of traditional image pyramids.
  3. Occlusion-aware Re-ID Strategy: By using pose estimation outputs to assess the visibility of keypoints, the system can determine if a Re-ID feature is reliable. If too many joints are hidden, the system adjusts its tracking logic to avoid errors during occlusions.
The FastPose architecture featuring a shared backbone and specialized heads for detection, pose, and Re-ID.

Experimental Results

FastPose was evaluated on standard benchmarks like COCO, MPII, and PoseTrack. The results demonstrate a superior balance between speed and accuracy compared to existing state-of-the-art methods.

  • Real-time Performance: With a ResNet-18 backbone, FastPose achieves 29.4 FPS, comfortably meeting real-time requirements. Even with a heavier ResNet-50 backbone, it maintains 12.2 FPS.
  • High Accuracy: The ResNet-101 variant reaches 70.3 mAP on the PoseTrack validation set, outperforming many complex, non-real-time models.
  • Robust Tracking: The occlusion-aware strategy reduces ID switches (where the tracker loses track of individuals) by 37%, proving the effectiveness of pose-guided tracking.
Accuracy (MOTA and mAP) vs. Speed (FPS) comparison of FastPose variants against other state-of-the-art methods.

Conclusion

FastPose demonstrates that a unified multi-task network can match the accuracy of complex two-stage systems while maintaining real-time speeds. By focusing on scale normalization and leveraging pose information for tracking, it provides a robust framework for dynamic video analysis. Future work may extend this approach to 3D pose estimation and more complex crowded environments.

Summary updated

Abstract

Both accuracy and efficiency are significant for pose estimation and tracking in videos. State-of-the-art performance is dominated by two-stages top-down methods. Despite the leading results, these methods are impractical for real-world applications due to their separated architectures and complicated calculation. This paper addresses the task of articulated multi-person pose estimation and tracking towards real-time speed. An end-to-end multi-task network (MTN) is designed to perform human detection, pose estimation, and person re-identification (Re-ID) tasks simultaneously. To alleviate the performance bottleneck caused by scale variation problem, a paradigm which exploits scale-normalized image and feature pyramids (SIFP) is proposed to boost both performance and speed. Given the results of MTN, we adopt an occlusion-aware Re-ID feature strategy in the pose tracking module, where pose information is utilized to infer the occlusion state to make better use of Re-ID feature. In experiments, we demonstrate that the pose estimation and tracking performance improves steadily utilizing SIFP through different backbones. Using ResNet-18 and ResNet-50 as backbones, the overall pose tracking framework achieves competitive performance with 29.4 FPS and 12.2 FPS, respectively. Additionally, occlusion-aware Re-ID feature decreases the identification switches by 37% in the pose tracking process.
BibTeX
@misc{zhang2019fastpose,
  author = {Jiabin Zhang and Zheng Zhu and Wei Zou and Peng Li and Yanwei Li and Hu Su and Guan Huang},
  title = {{FastPose: Towards Real-time Pose Estimation and Tracking via Scale-normalized Multi-task Networks}},
  year = {2019},
  eprint = {1908.05593},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.05593},
}
Figure