Abstract
Multi-person pose estimation from a 2D image is an essential technique for human behavior understanding. In this paper, we propose a human pose refinement network that estimates a refined pose from a tuple of an input image and input pose. The pose refinement was performed mainly through an end-to-end trainable multi-stage architecture in previous methods. However, they are highly dependent on pose estimation models and require careful model design. By contrast, we propose a model-agnostic pose refinement method. According to a recent study, state-of-the-art 2D human pose estimation methods have similar error distributions. We use this error statistics as prior information to generate synthetic poses and use the synthesized poses to train our model. In the testing stage, pose estimation results of any other methods can be input to the proposed method. Moreover, the proposed model does not require code or knowledge about other methods, which allows it to be easily used in the post-processing step. We show that the proposed approach achieves better performance than the conventional multi-stage refinement models and consistently improves the performance of various state-of-the-art pose estimation methods on the commonly used benchmark. The code is available in (https://github.com/mks0601/PoseFix_RELEASE).
Introduction
Human pose estimation—the task of identifying keypoints like elbows, knees, and eyes in an image—has seen massive gains thanks to Deep Learning. However, even the best models still make mistakes. These errors usually fall into specific categories: "jitter" (slight misalignments), "inversion" (confusing left and right limbs), or "swaps" (assigning a keypoint to the wrong person).
Traditionally, researchers tried to fix these errors using "multi-stage" architectures, where the network refines its own predictions in successive passes. The problem is that these refinement stages are tightly coupled to the specific model they were built for. PoseFix addresses this by decoupling the refiner from the estimator, creating a general-purpose "pose corrector" that can take the output of any human pose estimation model and make it better.
Methods
The core innovation of PoseFix is its model-agnostic training strategy. Instead of training the refiner on the outputs of a specific existing model, the authors use ground-truth data to synthesize "broken" poses.
Synthetic Error Generation
Using error statistics from a taxonomy of common failures (jitter, inversion, swap, and miss), the authors take perfect ground-truth poses and intentionally break them. For example, they swap left-side keypoints with right-side ones (inversion) or replace a person's keypoint with one from a nearby person (swap). This allows the model to learn a general mapping from erroneous poses to correct ones.
Coarse-to-Fine Architecture
PoseFix takes a tuple as input: the original image and the "coarse" input pose, represented as Gaussian heatmaps. These heatmaps act like an attention mechanism, telling the network where to look.
The network (using a ResNet backbone) produces a refined heatmap and uses a soft-argmax operation to extract high-precision coordinates. This allows the model to output coordinates in a continuous space, avoiding the quantization errors common in standard heatmap methods. Because PoseFix doesn't need to know how the initial pose was generated, it can be dropped into any existing pipeline as a post-processing step.
Key Results
PoseFix consistently improved every model it was tested against on the MS COCO 2017 Keypoint Detection dataset. Some highlights from the MS COCO test-dev set include:
- CPN (ResNet-50): Improved from 68.6 to 71.8 AP (+3.2 gain).
- SimpleBaseline (ResNet-152): Improved from 71.1 to 73.6 AP (+2.5 gain).
- Mask R-CNN: Showed significant gains across various backbones (e.g., +4.3 AP for ResNet-50).
Qualitatively, the model was particularly effective at fixing "inversion" errors and "swaps" in crowded scenes.
Conclusion
PoseFix demonstrates that a model-agnostic approach to pose refinement is not only possible but highly effective. By leveraging shared error distributions across different models, it provides a universal tool for boosting accuracy in human pose estimation tasks. Future work could extend this approach to 3D pose refinement or incorporating temporal consistency for video-based estimation.
Summary updated