Person Image Synthesis via Denoising Diffusion Model

Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, Fahad Shahbaz Khan

Introduction

Pose-guided person image synthesis is a fundamental task in computer vision, with applications ranging from virtual try-ons to movie production. The objective is to generate a photorealistic image of a person in a specific target pose, given a source image of that person. Traditionally, this field has been dominated by Generative Adversarial Networks (GANs). While GANs have made significant strides, they often struggle with complex spatial transformations, leading to distorted textures, unrealistic body shapes, and artifacts in occluded regions. These limitations stem from the "one-shot" nature of GAN generation, which finds it difficult to map complex deformations in a single step.

To overcome these challenges, the authors propose the Person Image Diffusion Model (PIDM). By leveraging the iterative denoising power of diffusion models, PIDM breaks down the complex transformation into a series of simpler, manageable steps, learning a "plausible trajectory" from noise to a high-fidelity person image.

Methodology: The PIDM Framework

PIDM treats person synthesis as a conditional denoising process. The model starts with pure Gaussian noise and iteratively refines it, guided by the source image's style and the target pose's skeleton.

Texture Diffusion Blocks (TDB)

A key innovation in PIDM is the Texture Diffusion Block (TDB). Standard diffusion models often fail to preserve fine-grained texture details from a source image during the denoising process. PIDM addresses this by using a dedicated texture encoder to extract multi-scale features from the source image. These features are then injected into the denoising UNet via a cross-attention mechanism. Unlike traditional flow-based warping methods that require explicit geometric alignment, cross-attention allows the model to dynamically "attend" to relevant source textures at every denoising step, ensuring high-fidelity reproduction of clothing patterns and skin details.

The PIDM architecture featuring Texture Diffusion Blocks (TDB) and Disentangled Classifier-Free Guidance.

Disentangled Classifier-Free Guidance (DCF)

In typical conditional diffusion models, multiple conditions are often combined into a single guidance signal. However, the authors observed that lumping pose and style together can lead to a "guidance conflict," where the model prioritizes one condition at the expense of the other. PIDM introduces Disentangled Classifier-Free Guidance (DCF), which separates the guidance for pose and appearance. By employing independent guidance scales ($w_p$ for pose and $w_s$ for style), the model can be precisely steered to satisfy both constraints simultaneously, resulting in better alignment with the input requirements.

Experimental Setup and Datasets

The model was rigorously evaluated on two benchmark datasets:

  1. DeepFashion (In-shop Clothes Retrieval): A high-resolution dataset (~52,000 images) featuring professional fashion models.
  2. Market-1501: A challenging low-resolution dataset (~32,000 images) from surveillance footage, characterized by diverse backgrounds and lighting conditions.

Results and Analysis

PIDM establishes a new state-of-the-art (SOTA) across both quantitative and qualitative metrics.

Quantitative Performance

On the DeepFashion benchmark ($256 \times 176$), PIDM achieved a Frechet Inception Distance (FID) of 6.36, a substantial improvement over the previous SOTA, NTED, which scored 8.68. Lower FID scores indicate higher image realism and better distribution matching.

Qualitative Comparison

Visually, PIDM produces images with significantly fewer artifacts than GAN-based competitors. As shown in the comparison below, PIDM excels at maintaining texture consistency even under extreme pose changes, where other models often produce "ghosting" or "stretching" effects.

Qualitative comparison of PIDM against SOTA GAN-based methods on the DeepFashion dataset.

User Study and Downstream Utility

In a human evaluation study involving 100 participants, PIDM was preferred 56% of the time (the "Jab" score), dwarfing the next best model's 18%. Furthermore, the authors demonstrated the practical utility of PIDM by using it to augment training data for person re-identification (re-ID). Adding 30,000 synthetic images to the Market-1501 training set improved the mean Average Precision (mAP) from 76.7% to 78.4%, proving that the generated images contain meaningful semantic information.

Conclusion and Future Work

PIDM represents a significant shift in person image synthesis, demonstrating that diffusion models can outperform GANs in handling complex spatial deformations. By replacing rigid warping with flexible cross-attention and disentangled guidance, PIDM achieves unprecedented realism. While inference speed remains a challenge common to diffusion models, the high quality of the results opens new doors for applications in virtual fashion and video synthesis. Future work will focus on optimizing sampling speed and enabling more granular control over individual garment components.

Summary updated

Abstract

The pose-guided person image generation task requires synthesizing photorealistic images of humans in arbitrary poses. The existing approaches use generative adversarial networks that do not necessarily maintain realistic textures or need dense correspondences that struggle to handle complex deformations and severe occlusions. In this work, we show how denoising diffusion models can be applied for high-fidelity person image synthesis with strong sample diversity and enhanced mode coverage of the learnt data distribution. Our proposed Person Image Diffusion Model (PIDM) disintegrates the complex transfer problem into a series of simpler forward-backward denoising steps. This helps in learning plausible source-to-target transformation trajectories that result in faithful textures and undistorted appearance details. We introduce a 'texture diffusion module' based on cross-attention to accurately model the correspondences between appearance and pose information available in source and target images. Further, we propose 'disentangled classifier-free guidance' to ensure close resemblance between the conditional inputs and the synthesized output in terms of both pose and appearance information. Our extensive results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios. We also show how our generated images can help in downstream tasks. Our code and models will be publicly released.
BibTeX
@inproceedings{bhunia2022person,
  author = {Ankan Kumar Bhunia and Salman Khan and Hisham Cholakkal and Rao Muhammad Anwer and Jorma Laaksonen and Mubarak Shah and Fahad Shahbaz Khan},
  title = {{Person Image Synthesis via Denoising Diffusion Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52729.2023.00578},
}
Figure