Focus and retain: Complement the Broken Pose in Human Image Synthesis

Pu Ge, Qiushi Huang, Wei Xiang, Xue Jing, Yule Li, Yiyong Li, Zhun SunWACV 2021 · OpenReview

Abstract

Given a target pose, how to generate an image of a specific style with that target pose remains an ill-posed and thus complicated problem. Most recent works treat the human pose synthesis tasks as an image spatial transformation problem using flow warping techniques. However, we observe that, due to the inherent ill-posed nature of many complicated human poses, former methods fail to generate body parts. To tackle this problem, we propose a feature-level flow attention module and an Enhancer Network. The flow attention module produces a flow attention mask to guide the combination of the flow-warped features and the structural pose features. Then, we apply the Enhancer Network to re-fine the coarse image by injecting the pose information. We present our experimental evaluation both qualitatively and quantitatively on DeepFashion, Market-1501, and Youtube dance datasets. Quantitative results show that our method has 12.995 FID at DeepFashion, 25.459 FID at Market-1501, 14.516 FID at Youtube dance datasets, which outperforms some state-of-the-arts including Guide-Pixe2Pixe, Global-Flow-Local-Attn, and CocosNet.

Introduction

Human pose transfer—taking an image of a person and "re-posing" them into a new target position—is a classic challenge in computer vision. Most modern methods rely on optical flow, a technique that "warps" pixels from the source image to the target pose.

However, this approach hits a wall when poses get complicated. If a person’s arm is hidden behind their back in the source image but extended in the target pose, the optical flow has no source pixels to "warp" from. This leads to "broken limbs" or blurry artifacts. Furthermore, as data passes through deep neural networks, the specific structural constraints of the human body often get "diluted," resulting in anatomically impossible generations.

Extracted figure from page 1 of Focus and retain: Complement the Broken Pose in Human Image Synthesis

Methods

The authors propose a two-stage "Focus and Retain" strategy:

  1. Flow-Attention Network (FAN): Instead of just warping pixels, this module generates a Flow-Attention Mask ($M$). This mask acts as a smart selector: for regions where the source image provides clear texture (like a torso), it uses warped features. For regions where the flow fails (like an occluded limb), it switches to using structural pose features to "synthesize" the missing part from scratch.
  2. Enhancer Network (EN): To prevent "information dilution," the authors add a refinement stage. This network uses SPADE (Spatially-Adaptive De-normalization) blocks. These blocks effectively "inject" the target pose map back into the generation process at multiple layers, ensuring the final image strictly adheres to the intended skeletal structure.
  3. Specialized Losses: The model is trained using a combination of standard reconstruction losses, adversarial (GAN) losses, and a specific Face Loss that focuses on six key facial landmarks to ensure the person's identity remains recognizable.

Experimental Evaluation

The model was evaluated on three distinct benchmarks: DeepFashion (high-resolution), Market-1501 (low-resolution surveillance), and YouTube-dance (complex high-motion). The authors also introduced a new metric, Pose Reconstruction Error (PRE), which uses a separate pose estimator (AlphaPose) on the generated images to see how closely the AI's output matches the intended target pose coordinates.

Key Results

The proposed method outperformed several state-of-the-art models (including CocosNet and GFLA) across all metrics:

  • DeepFashion: Achieved an FID of 12.995 (lower is better), compared to GFLA's 13.234.
  • YouTube-dance: Achieved an FID of 14.516, significantly beating the competition.
  • Subjective Quality: In "Just Noticeable Difference" (JND) tests, human volunteers preferred the authors' model's results roughly 26.8% of the time on DeepFashion, nearly double the preference for previous top-tier methods.
Figure 5: Detailed comparison of generated results highlighting the "broken limb" problem in previous methods (center) versus the proposed approach (right).

Conclusion and Future Work

By recognizing that flow-based warping is excellent for texture but poor for structure during occlusions, the "Focus and Retain" strategy effectively hybridizes these approaches. The use of SPADE blocks ensures that the target pose guides the entire synthesis process, preventing anatomical dilution. While the model significantly improves limb consistency, extreme occlusions of complex textures remain a challenge. Future work may focus on further refining the pose injection process for enhanced temporal consistency in video synthesis.

Summary updated

Abstract

Given a target pose, how to generate an image of a specific style with that target pose remains an ill-posed and thus complicated problem. Most recent works treat the human pose synthesis tasks as an image spatial transformation problem using flow warping techniques. However, we observe that, due to the inherent ill-posed nature of many complicated human poses, former methods fail to generate body parts. To tackle this problem, we propose a feature-level flow attention module and an Enhancer Network. The flow attention module produces a flow attention mask to guide the combination of the flow-warped features and the structural pose features. Then, we apply the Enhancer Network to re-fine the coarse image by injecting the pose information. We present our experimental evaluation both qualitatively and quantitatively on DeepFashion, Market-1501, and Youtube dance datasets. Quantitative results show that our method has 12.995 FID at DeepFashion, 25.459 FID at Market-1501, 14.516 FID at Youtube dance datasets, which outperforms some state-of-the-arts including Guide-Pixe2Pixe, Global-Flow-Local-Attn, and CocosNet.
BibTeX
@inproceedings{ge2021focus,
  author = {Pu Ge and Qiushi Huang and Wei Xiang and Xue Jing and Yule Li and Yiyong Li and Zhun Sun},
  title = {{Focus and retain: Complement the Broken Pose in Human Image Synthesis}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=cAP2Z5MLtNO},
}
Figure