Abstract
Top-down approaches for human pose estimation (HPE) have reached a high level of sophistication, exemplified by models such as HRNet and ViTPose. Nonetheless, the low efficiency of top-down methods is a recognized issue that has not been sufficiently explored in current research. Our analysis suggests that the primary cause of inefficiency stems from the substantial diversity found in pose samples. Simple poses can be accurately estimated without requiring the computational resources of larger models, while an abundance of redundant bounding boxes from detectors adds unnecessary load. In this paper, we present a straightforward yet effective dynamic framework called DynPose, designed to match diverse pose samples with the most appropriate models. Specifically, the framework contains a lightweight router and two pre-trained HPE models: one small and one large. Extensive experiments demonstrate that DynPose achieves an almost 50% increase in speed over HRNet-W32 while maintaining the same-level accuracy, with strong generalization across different models and datasets.
Introduction: The Efficiency Bottleneck in HPE
Top-down human pose estimation (HPE) is currently the gold standard for accuracy. It typically follows a two-step pipeline: an object detector identifies people in an image, and a pose estimation model predicts keypoints for every detected box. However, this approach faces a significant efficiency challenge. Because computational cost scales linearly with the number of detected boxes, inference becomes prohibitively slow in crowded scenes or when detectors produce redundant, low-quality boxes.
Current research often focuses on making individual models smaller, but this ignores a fundamental truth: not all pose samples are equally difficult. Treating a clear, simple pose with the same heavy-duty model used for a heavily occluded one is a massive waste of resources. DynPose addresses this by introducing a dynamic routing mechanism that adapts computational effort to the complexity of the input.
The DynPose Framework
DynPose is a "plug-and-play" framework that can be integrated with existing pre-trained models. It consists of three primary components: a lightweight Router, a Small Network, and a Large Network.
Dynamic Routing Strategy
Instead of a simple binary choice, the router (a tiny CNN costing only 0.19G FLOPs) categorizes input crops into three distinct paths:
- Easy Samples: These are clear, high-quality poses. They are sent to the Small Network because the accuracy gain from a larger model is negligible.
- Hard Samples: These are challenging poses where high-capacity models provide the most benefit. These are sent to the Large Network.
- Extremely Hard Samples: These are often "garbage" boxes—redundant detections or poor crops where even the Large Network fails. Counter-intuitively, DynPose routes these back to the Small Network. This saves significant computation on samples that would likely result in low accuracy regardless of model size.
To train the router effectively, the authors utilize a novel Object Keypoint Distance (OKD) loss. Unlike standard metrics that saturate quickly, OKD better reflects the actual performance gap between the small and large models, providing a clearer signal for the router to learn sample difficulty.
Experimental Results
DynPose was primarily evaluated on the COCO 2017 Keypoint Detection benchmark and tested for generalization on the CrowdPose dataset.
Efficiency vs. Accuracy
When using ResNet-50 as the small model and HRNet-W32 as the large model, DynPose achieved a 50% speedup over a standalone HRNet-W32 while maintaining identical accuracy on COCO. The framework scales effectively to state-of-the-art architectures as well; when paired with ViTPose-L, it maintained a 78.0% AP while increasing the frame rate from 59 to 93 FPS and reducing FLOPs by nearly 40%.
Generalization and Decoupled Training
A key advantage of DynPose is its decoupled nature. The pre-trained pose models do not need to be retrained; only the lightweight router is optimized. Furthermore, a router trained on one pair of models (e.g., ResNet and HRNet) can be applied to entirely different architectures (e.g., MobileNet and ViTPose) in a zero-shot manner, maintaining effective performance trade-offs.
Conclusion
DynPose demonstrates that accounting for input diversity is a powerful lever for improving the efficiency of top-down human pose estimation. By intelligently routing "easy" and "extremely hard" samples to lightweight models, it achieves significant speedups without sacrificing accuracy. This dynamic approach is highly generalizable and offers a promising path for other computationally intensive computer vision tasks.
Summary updated