Introduction
Traditional human pose estimation has largely been fragmented, focusing independently on body joints, facial landmarks, or hand gestures. To achieve a comprehensive "whole-body" understanding, researchers previously relied on "Frankenstein" ensembles—stitching together multiple specialized models. However, these multi-model approaches suffer from dataset bias, high computational complexity, and significant scale variance issues, particularly when resolving tiny hand joints alongside large body structures.
The COCO-WholeBody Dataset
To address the lack of unified data, the authors introduce COCO-WholeBody, the first large-scale benchmark for 2D whole-body pose estimation. Extending the original COCO dataset, it provides 133 manual landmark annotations (68 for the face, 42 for hands, and 23 for the body and feet) across 200,000 images. Unlike lab-captured datasets, COCO-WholeBody captures "in the wild" scenarios featuring complex poses, motion blur, and occlusion.
The ZoomNet Architecture
The proposed ZoomNet is a top-down, end-to-end trainable network designed to handle the massive scale differences between body parts. It employs a hierarchical strategy:
- FeatureNet: Extracts shared low-level features.
- BodyNet: Predicts standard body/foot keypoints and identifies bounding boxes for hands and faces.
- Zooming (ROIAlign): The network "zooms in" by extracting high-resolution features from these predicted regions.
- Specialized Heads: Dedicated FaceHead and HandHead sub-networks process the zoomed features to localize fine-grained landmarks.
By unifying these tasks, ZoomNet learns spatial correlations (e.g., the hand's position relative to the elbow) while maintaining the resolution necessary for small extremities.
Experimental Results
ZoomNet demonstrates a significant performance leap over existing methods. It achieves a 0.541 mAP on the whole-body task, substantially outperforming the previous state-of-the-art, OpenPose (0.338 mAP).
Beyond accuracy, ZoomNet is remarkably efficient. It requires only 27.36G FLOPs, compared to the 451.09G FLOPs needed by the OpenPose whole-body ensemble—a 16x reduction in computational cost. Furthermore, pre-training on COCO-WholeBody was found to improve performance on specialized tasks like facial landmark detection (300W) and hand pose estimation (Panoptic), suggesting that holistic body context aids part-level localization.
Conclusion
The introduction of COCO-WholeBody and ZoomNet marks a shift toward unified, efficient whole-body pose estimation. While challenges remain regarding extreme motion blur and very low-resolution instances, ZoomNet provides a robust foundation for future applications in 3D mesh recovery and real-time human-computer interaction.
Summary updated