Introduction
Pose-guided person image synthesis is a fundamental task in computer vision, with applications ranging from virtual try-ons to movie production. The objective is to generate a photorealistic image of a person in a specific target pose, given a source image of that person. Traditionally, this field has been dominated by Generative Adversarial Networks (GANs). While GANs have made significant strides, they often struggle with complex spatial transformations, leading to distorted textures, unrealistic body shapes, and artifacts in occluded regions. These limitations stem from the "one-shot" nature of GAN generation, which finds it difficult to map complex deformations in a single step.
To overcome these challenges, the authors propose the Person Image Diffusion Model (PIDM). By leveraging the iterative denoising power of diffusion models, PIDM breaks down the complex transformation into a series of simpler, manageable steps, learning a "plausible trajectory" from noise to a high-fidelity person image.
Methodology: The PIDM Framework
PIDM treats person synthesis as a conditional denoising process. The model starts with pure Gaussian noise and iteratively refines it, guided by the source image's style and the target pose's skeleton.
Texture Diffusion Blocks (TDB)
A key innovation in PIDM is the Texture Diffusion Block (TDB). Standard diffusion models often fail to preserve fine-grained texture details from a source image during the denoising process. PIDM addresses this by using a dedicated texture encoder to extract multi-scale features from the source image. These features are then injected into the denoising UNet via a cross-attention mechanism. Unlike traditional flow-based warping methods that require explicit geometric alignment, cross-attention allows the model to dynamically "attend" to relevant source textures at every denoising step, ensuring high-fidelity reproduction of clothing patterns and skin details.
Disentangled Classifier-Free Guidance (DCF)
In typical conditional diffusion models, multiple conditions are often combined into a single guidance signal. However, the authors observed that lumping pose and style together can lead to a "guidance conflict," where the model prioritizes one condition at the expense of the other. PIDM introduces Disentangled Classifier-Free Guidance (DCF), which separates the guidance for pose and appearance. By employing independent guidance scales ($w_p$ for pose and $w_s$ for style), the model can be precisely steered to satisfy both constraints simultaneously, resulting in better alignment with the input requirements.
Experimental Setup and Datasets
The model was rigorously evaluated on two benchmark datasets:
- DeepFashion (In-shop Clothes Retrieval): A high-resolution dataset (~52,000 images) featuring professional fashion models.
- Market-1501: A challenging low-resolution dataset (~32,000 images) from surveillance footage, characterized by diverse backgrounds and lighting conditions.
Results and Analysis
PIDM establishes a new state-of-the-art (SOTA) across both quantitative and qualitative metrics.
Quantitative Performance
On the DeepFashion benchmark ($256 \times 176$), PIDM achieved a Frechet Inception Distance (FID) of 6.36, a substantial improvement over the previous SOTA, NTED, which scored 8.68. Lower FID scores indicate higher image realism and better distribution matching.
Qualitative Comparison
Visually, PIDM produces images with significantly fewer artifacts than GAN-based competitors. As shown in the comparison below, PIDM excels at maintaining texture consistency even under extreme pose changes, where other models often produce "ghosting" or "stretching" effects.
User Study and Downstream Utility
In a human evaluation study involving 100 participants, PIDM was preferred 56% of the time (the "Jab" score), dwarfing the next best model's 18%. Furthermore, the authors demonstrated the practical utility of PIDM by using it to augment training data for person re-identification (re-ID). Adding 30,000 synthetic images to the Market-1501 training set improved the mean Average Precision (mAP) from 76.7% to 78.4%, proving that the generated images contain meaningful semantic information.
Conclusion and Future Work
PIDM represents a significant shift in person image synthesis, demonstrating that diffusion models can outperform GANs in handling complex spatial deformations. By replacing rigid warping with flexible cross-attention and disentangled guidance, PIDM achieves unprecedented realism. While inference speed remains a challenge common to diffusion models, the high quality of the results opens new doors for applications in virtual fashion and video synthesis. Future work will focus on optimizing sampling speed and enabling more granular control over individual garment components.
Summary updated