Abstract
Human pose estimation is of critical importance for visual understanding tasks such as action recognition and human-computer interaction. In this work, we present a Multiple Stage High-Resolution Network (Multi-Stage HRNet) to tackle the problem of multi-person pose estimation in images. Specifically, we follow top-down pipelines where high-resolution representations are maintained during single-person pose estimation. In addition, a multiple stage network and cross-stage feature aggregation are adopted to further refine keypoint positions. The resulting approach achieves promising results on the COCO dataset. Our single-model-single-scale test configuration obtains a 77.1 AP score on the test-dev set using publicly available training data.
Introduction
Identifying the (x, y) coordinates of human joints—such as elbows, knees, and ankles—is a foundational task in computer vision. Despite significant progress, "in-the-wild" scenarios characterized by heavy occlusion, intense lighting, and complex body poses remain challenging.
Most traditional top-down approaches utilize encoder-decoder architectures that downsample images to a low-resolution bottleneck before upsampling. This process often results in the loss of fine-grained spatial information. Multi-Stage HRNet addresses this by maintaining high-resolution representations throughout the network while implementing a multi-stage refinement mechanism to improve initial predictions.
Methods
The Multi-Stage HRNet architecture is built upon three core pillars:
- HRNet Backbone: Unlike networks that downsample to a bottleneck, HRNet maintains a high-resolution stream in parallel with lower-resolution streams. These streams undergo continuous multi-scale fusion, ensuring the network preserves precise spatial details alongside global context.
- Multi-Stage Cascading: The model stacks two HRNet units in sequence. The first stage generates an initial pose estimation, which the second stage then refines. While more stages could theoretically improve results, the authors utilized two stages to balance performance with GPU memory constraints.
- Cross-Stage Feature Aggregation: This mechanism allows the second stage to "inherit" rich features from the first. Rather than just passing the final heatmap, intermediate feature maps from every resolution level in Stage 1 are added to the corresponding levels in Stage 2 via element-wise operations.
Datasets and Evaluation
The researchers evaluated the model using several major benchmarks to ensure robustness across different scenarios:
- COCO (Common Objects in Context): The primary benchmark for keypoint detection.
- MPII Human Pose: A standard dataset for single-person pose estimation.
- AI Challenger: A large-scale dataset used to provide "extra data" for performance boosting.
- CrowdPose: Designed to test pose estimation in crowded scenes with significant person-to-person overlap.
Key Results and Insights
The proposed Multi-Stage HRNet demonstrates that maintaining high-resolution representations is inherently superior for localization tasks compared to traditional "hourglass" downsampling approaches.
Performance on COCO
Using a single-model, single-scale test configuration, the model achieved 77.1 AP on the COCO test-dev benchmark. When utilizing model ensembles and test-time augmentation, the performance reached 77.7 AP, ranking 2nd on the COCO leaderboard at the time of publication.
Ablation and Data Impact
Ablation studies confirmed that adding the second stage provided a consistent boost of 0.3 to 0.4 AP across various backbone sizes. Furthermore, the inclusion of AI Challenger and CrowdPose datasets in the training phase was a major driver of performance, jumping from 75.8 to 77.1 AP.
Conclusion
Multi-Stage HRNet successfully combines the spatial precision of HRNet with the iterative refinement of multi-stage networks. The cross-stage feature aggregation ensures that refinement stages have access to raw spatial evidence, preventing gradient issues and enabling finer keypoint adjustments. Future work may explore expanding the number of stages as hardware allows and integrating temporal tracking for video-based pose estimation.
Summary updated