Multi-Stage HRNet: Multiple Stage High-Resolution Network for Human Pose Estimation

Junjie Huang, Zheng Zhu, Guan HuangarXiv 2019 · OpenReview

Abstract

Human pose estimation is of critical importance for visual understanding tasks such as action recognition and human-computer interaction. In this work, we present a Multiple Stage High-Resolution Network (Multi-Stage HRNet) to tackle the problem of multi-person pose estimation in images. Specifically, we follow top-down pipelines where high-resolution representations are maintained during single-person pose estimation. In addition, a multiple stage network and cross-stage feature aggregation are adopted to further refine keypoint positions. The resulting approach achieves promising results on the COCO dataset. Our single-model-single-scale test configuration obtains a 77.1 AP score on the test-dev set using publicly available training data.

Introduction

Identifying the (x, y) coordinates of human joints—such as elbows, knees, and ankles—is a foundational task in computer vision. Despite significant progress, "in-the-wild" scenarios characterized by heavy occlusion, intense lighting, and complex body poses remain challenging.

Most traditional top-down approaches utilize encoder-decoder architectures that downsample images to a low-resolution bottleneck before upsampling. This process often results in the loss of fine-grained spatial information. Multi-Stage HRNet addresses this by maintaining high-resolution representations throughout the network while implementing a multi-stage refinement mechanism to improve initial predictions.

Methods

The Multi-Stage HRNet architecture is built upon three core pillars:

  1. HRNet Backbone: Unlike networks that downsample to a bottleneck, HRNet maintains a high-resolution stream in parallel with lower-resolution streams. These streams undergo continuous multi-scale fusion, ensuring the network preserves precise spatial details alongside global context.
  2. Multi-Stage Cascading: The model stacks two HRNet units in sequence. The first stage generates an initial pose estimation, which the second stage then refines. While more stages could theoretically improve results, the authors utilized two stages to balance performance with GPU memory constraints.
  3. Cross-Stage Feature Aggregation: This mechanism allows the second stage to "inherit" rich features from the first. Rather than just passing the final heatmap, intermediate feature maps from every resolution level in Stage 1 are added to the corresponding levels in Stage 2 via element-wise operations.
The Multi-Stage HRNet architecture, illustrating the two-stage cascade and cross-stage feature aggregation via element-wise addition.

Datasets and Evaluation

The researchers evaluated the model using several major benchmarks to ensure robustness across different scenarios:

  • COCO (Common Objects in Context): The primary benchmark for keypoint detection.
  • MPII Human Pose: A standard dataset for single-person pose estimation.
  • AI Challenger: A large-scale dataset used to provide "extra data" for performance boosting.
  • CrowdPose: Designed to test pose estimation in crowded scenes with significant person-to-person overlap.

Key Results and Insights

The proposed Multi-Stage HRNet demonstrates that maintaining high-resolution representations is inherently superior for localization tasks compared to traditional "hourglass" downsampling approaches.

Performance on COCO

Using a single-model, single-scale test configuration, the model achieved 77.1 AP on the COCO test-dev benchmark. When utilizing model ensembles and test-time augmentation, the performance reached 77.7 AP, ranking 2nd on the COCO leaderboard at the time of publication.

Comparison of Multi-Stage HRNet with other state-of-the-art methods on the COCO test-dev dataset.

Ablation and Data Impact

Ablation studies confirmed that adding the second stage provided a consistent boost of 0.3 to 0.4 AP across various backbone sizes. Furthermore, the inclusion of AI Challenger and CrowdPose datasets in the training phase was a major driver of performance, jumping from 75.8 to 77.1 AP.

Conclusion

Multi-Stage HRNet successfully combines the spatial precision of HRNet with the iterative refinement of multi-stage networks. The cross-stage feature aggregation ensures that refinement stages have access to raw spatial evidence, preventing gradient issues and enabling finer keypoint adjustments. Future work may explore expanding the number of stages as hardware allows and integrating temporal tracking for video-based pose estimation.

Summary updated

Abstract

Human pose estimation are of importance for visual understanding tasks such as action recognition and human-computer interaction. In this work, we present a Multiple Stage High-Resolution Network (Multi-Stage HRNet) to tackling the problem of multi-person pose estimation in images. Specifically, we follow the top-down pipelines and high-resolution representations are maintained during single-person pose estimation. In addition, multiple stage network and cross stage feature aggregation are adopted to further refine the keypoint position. The resulting approach achieves promising results in COCO datasets. Our single-model-single-scale test configuration obtains 77.1 AP score in test-dev using publicly available training data.
BibTeX
@misc{huang2019multi,
  author = {Junjie Huang and Zheng Zhu and Guan Huang},
  title = {{Multi-Stage HRNet: Multiple Stage High-Resolution Network for Human Pose Estimation}},
  year = {2019},
  eprint = {1910.05901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.05901},
}
Figure