PoseFix: Model-Agnostic General Human Pose Refinement Network.

Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee

Abstract

Multi-person pose estimation from a 2D image is an essential technique for human behavior understanding. In this paper, we propose a human pose refinement network that estimates a refined pose from a tuple of an input image and input pose. The pose refinement was performed mainly through an end-to-end trainable multi-stage architecture in previous methods. However, they are highly dependent on pose estimation models and require careful model design. By contrast, we propose a model-agnostic pose refinement method. According to a recent study, state-of-the-art 2D human pose estimation methods have similar error distributions. We use this error statistics as prior information to generate synthetic poses and use the synthesized poses to train our model. In the testing stage, pose estimation results of any other methods can be input to the proposed method. Moreover, the proposed model does not require code or knowledge about other methods, which allows it to be easily used in the post-processing step. We show that the proposed approach achieves better performance than the conventional multi-stage refinement models and consistently improves the performance of various state-of-the-art pose estimation methods on the commonly used benchmark. The code is available in (https://github.com/mks0601/PoseFix_RELEASE).

Introduction

Human pose estimation—the task of identifying keypoints like elbows, knees, and eyes in an image—has seen massive gains thanks to Deep Learning. However, even the best models still make mistakes. These errors usually fall into specific categories: "jitter" (slight misalignments), "inversion" (confusing left and right limbs), or "swaps" (assigning a keypoint to the wrong person).

Traditionally, researchers tried to fix these errors using "multi-stage" architectures, where the network refines its own predictions in successive passes. The problem is that these refinement stages are tightly coupled to the specific model they were built for. PoseFix addresses this by decoupling the refiner from the estimator, creating a general-purpose "pose corrector" that can take the output of any human pose estimation model and make it better.

The PoseFix pipeline: an input pose and image are processed through a refinement network to produce a high-precision refined pose.

Methods

The core innovation of PoseFix is its model-agnostic training strategy. Instead of training the refiner on the outputs of a specific existing model, the authors use ground-truth data to synthesize "broken" poses.

Synthetic Error Generation

Using error statistics from a taxonomy of common failures (jitter, inversion, swap, and miss), the authors take perfect ground-truth poses and intentionally break them. For example, they swap left-side keypoints with right-side ones (inversion) or replace a person's keypoint with one from a nearby person (swap). This allows the model to learn a general mapping from erroneous poses to correct ones.

Coarse-to-Fine Architecture

PoseFix takes a tuple as input: the original image and the "coarse" input pose, represented as Gaussian heatmaps. These heatmaps act like an attention mechanism, telling the network where to look.

PoseFix is designed to be model-agnostic, accepting results from any estimation method.

The network (using a ResNet backbone) produces a refined heatmap and uses a soft-argmax operation to extract high-precision coordinates. This allows the model to output coordinates in a continuous space, avoiding the quantization errors common in standard heatmap methods. Because PoseFix doesn't need to know how the initial pose was generated, it can be dropped into any existing pipeline as a post-processing step.

Key Results

PoseFix consistently improved every model it was tested against on the MS COCO 2017 Keypoint Detection dataset. Some highlights from the MS COCO test-dev set include:

  • CPN (ResNet-50): Improved from 68.6 to 71.8 AP (+3.2 gain).
  • SimpleBaseline (ResNet-152): Improved from 71.1 to 73.6 AP (+2.5 gain).
  • Mask R-CNN: Showed significant gains across various backbones (e.g., +4.3 AP for ResNet-50).

Qualitatively, the model was particularly effective at fixing "inversion" errors and "swaps" in crowded scenes.

Conclusion

PoseFix demonstrates that a model-agnostic approach to pose refinement is not only possible but highly effective. By leveraging shared error distributions across different models, it provides a universal tool for boosting accuracy in human pose estimation tasks. Future work could extend this approach to 3D pose refinement or incorporating temporal consistency for video-based estimation.

Summary updated

Abstract

Multi-person pose estimation from a 2D image is an essential technique for human behavior understanding. In this paper, we propose a human pose refinement network that estimates a refined pose from a tuple of an input image and input pose. The pose refinement was performed mainly through an end-to-end trainable multi-stage architecture in previous methods. However, they are highly dependent on pose estimation models and require careful model design. By contrast, we propose a model-agnostic pose refinement method. According to a recent study, state-of-the-art 2D human pose estimation methods have similar error distributions. We use this error statistics as prior information to generate synthetic poses and use the synthesized poses to train our model. In the testing stage, pose estimation results of any other methods can be input to the proposed method. Moreover, the proposed model does not require code or knowledge about other methods, which allows it to be easily used in the post-processing step. We show that the proposed approach achieves better performance than the conventional multi-stage refinement models and consistently improves the performance of various state-of-the-art pose estimation methods on the commonly used benchmark. The code is available in (https://github.com/mks0601/PoseFix_RELEASE).
BibTeX
@inproceedings{moon2019posefix,
  author = {Gyeongsik Moon and Ju Yong Chang and Kyoung Mu Lee},
  title = {{PoseFix: Model-Agnostic General Human Pose Refinement Network.}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=rm-v0MxdTB},
}
Figure