Differentiable Rendering-Based Pose-Conditioned Human Image Generation

Yusuke Horiuchi, Edgar Simo-Serra, Satoshi Iizuka, Hiroshi IshikawaWorkshops @ CVPR 2021 · DOI

Introduction

Generating realistic images of humans in specific target poses, a task known as pose transfer, remains a significant challenge in computer vision. The problem is inherently ill-defined because movement often leads to complex occlusions and changes in visible body parts. Traditional methods typically rely on 2D keypoint heatmaps generated through fixed heuristics, such as Gaussian blobs of a standard size. However, these "one-size-fits-all" approaches often fail to account for variations in body shape and image resolution, resulting in synthesized images with distorted limbs and unrealistic proportions.

This paper proposes a novel approach by introducing a differentiable vertex and edge renderer. By replacing static pose heatmaps with a trainable rendering module, the system can learn optimal parameters for joint size and limb thickness directly from data, leading to more accurate and coherent body shape synthesis.

Methodology

The proposed architecture is built upon the MS-DefGAN (Multiple-Source Deformable GAN) framework, which is designed to synthesize images from one or more reference sources. The core innovation lies in the replacement of standard pose inputs with two specialized, differentiable modules:

  1. Vertex Renderer: Instead of using a fixed standard deviation ($\sigma$) for Gaussian blobs at each joint, the model learns an optimal $\sigma$ for every individual vertex through backpropagation.
  2. Edge Renderer: This module renders the "bones" connecting the joints using learnable parameters for edge elongation ($\alpha$) and thickness ($\beta$). To maintain differentiability, the authors utilize a "ramped rectangular function" (rrect), which ensures smooth gradients at the edges of rendered shapes.
Overview of the proposed pose-conditioned human image generation architecture.

By integrating the pose representation into the end-to-end optimization process, the network learns a meaningful spatial prior that better informs the generator about the target person's physical structure.

Experiments and Results

The model was evaluated on two prominent benchmarks: Market-1501, featuring low-resolution surveillance images, and DeepFashion, which contains high-resolution fashion photography. Initial 2D keypoints were estimated using OpenPose.

Quantitative Performance

The differentiable rendering approach yielded consistent improvements over existing baselines such as VUnet and the original MS-DefGAN. On the Market-1501 dataset with a single reference image, the model achieved an SSIM of 0.324, compared to 0.318 for MS-DefGAN. Similar performance gains were observed on the DeepFashion dataset.

Qualitative Analysis

Visual evaluations confirm that the model produces more coherent body shapes. The learnable limb thickness allows for a better "spatial prior," particularly in complex regions like the fit of clothing or the orientation of limbs. Furthermore, the model scales effectively with multiple reference images ($M_n = 3$ or $5$), which helps resolve ambiguities and enhances output quality.

Qualitative comparison on the Market-1501 dataset across different numbers of source images ($M_n$).

Conclusion

While the quantitative improvements in metrics like SSIM are relatively modest, the qualitative results demonstrate a clear advantage in generating realistic human proportions. The primary contribution—moving from hardcoded heuristics to a learnable, differentiable pose representation—provides a more informative condition for GAN-based synthesis. Future work may involve extending this 2D skeleton-based approach to incorporate 3D body volume and clothing physics for even greater realism.

Summary updated

BibTeX
@inproceedings{horiuchi2021differentiable,
  author = {Yusuke Horiuchi and Edgar Simo-Serra and Satoshi Iizuka and Hiroshi Ishikawa},
  title = {{Differentiable Rendering-Based Pose-Conditioned Human Image Generation}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00437},
}
Figure