Spatial Content Alignment For Pose Transfer

Wing Yin Yu, Lai-Man Po, Yuzhi Zhao, Jingjing Xiong, Kin Wai Lau

Abstract

Due to unreliable geometric matching and content misalignment, most conventional pose transfer algorithms fail to generate fine-trained person images. In this paper, we propose a novel framework Spatial Content Alignment GAN (SCA-GAN) which aims to enhance the content consistency of garment textures and the details of human characteristics. We first alleviate the spatial misalignment by transferring the edge content to the target pose in advance. Secondly, we introduce a new Content-Style DeBlk which can progressively synthesize photo-realistic person images based on the appearance features of the source image, the target pose heatmap and the prior transferred content in edge domain. We compare the proposed framework with several state-of-the-art methods to show its superiority in quantitative and qualitative analysis. Moreover, detailed ablation study results demonstrate the efficacy of our contributions.

Introduction

Pose transfer—the task of synthesizing an image of a person in a new pose based on a source image—is notoriously difficult because the source and target images are "spatially misaligned." Existing methods generally fall into two camps: warping-based methods that attempt to geometrically stretch the source image, and attention-based methods that use neural networks to attend to relevant source features. However, both often fail to maintain fine-grained details like garment patterns and facial identity due to a lack of reliable spatial guidance.

Methodology

The proposed Spatial Content Alignment GAN (SCA-GAN) addresses these challenges through a two-phase framework that prioritizes structural alignment before final synthesis.

Phase 1: Prior Content Transfer Network (PCT-Net)

Instead of synthesizing pixels directly, the model first predicts the person's edges in the target pose. Edges represent high-frequency signals that capture the "skeleton" of both the person's identity and their clothing. The PCT-Net takes the source edge map, source pose, and target pose to generate a spatially aligned "blueprint" for the next phase.

Phase 2: Image Synthesis Network (IS-Net)

This network takes the source image and the predicted edges from Phase 1 to create the final photo. It features two novel components:

  • Style Encoder Blocks (Style EnBlk): These blocks omit standard normalization layers to preserve "unique pattern statistics," such as specific logos or intricate textures that might otherwise be smoothed out by normalization.
  • Content-Style SPADE (CS-SPADE): An evolution of the Spatially-Adaptive Denormalization technique, CS-SPADE simultaneously injects aligned content (target pose and predicted edges) and misaligned style (source image features) during the upsampling process.
The architecture of SCA-GAN, illustrating the two-phase approach: PCT-Net for edge transfer and IS-Net for image synthesis.

Experiments and Results

The researchers evaluated SCA-GAN on the DeepFashion dataset. Quantitatively, SCA-GAN outperformed several state-of-the-art models (including PATN, ADGAN, and APS), achieving the highest Inception Score (IS) of 3.497 and the lowest FID (11.676), indicating superior image quality and perceptual similarity to real photos.

Qualitatively, visual results show much sharper clothing boundaries, realistic creases in garments, and better preservation of facial features compared to competitors, which often produced "blobby" or distorted results in those areas. Ablation studies confirmed that the edge-prediction step is crucial; removing the prior edge transfer caused the FID score to jump from 11.6 to 12.4.

Qualitative comparison showing SCA-GAN's superior ability to preserve textures and details compared to state-of-the-art methods like PATN, APS, and ADGAN.

Conclusion

SCA-GAN effectively solves the spatial misalignment problem in pose transfer by utilizing edge maps as an intermediate spatial guide. By decoupling structural content from appearance style, the framework achieves high-fidelity synthesis of person images with consistent textures and sharp details. Codes are publicly available at github.com/rocketappslab/SCA-GAN.

Summary updated

Abstract

Due to unreliable geometric matching and content misalignment, most conventional pose transfer algorithms fail to generate fine-trained person images. In this paper, we propose a novel framework Spatial Content Alignment GAN (SCAGAN) which aims to enhance the content consistency of garment textures and the details of human characteristics. We first alleviate the spatial misalignment by transferring the edge content to the target pose in advance. Secondly, we introduce a new Content-Style DeBlk which can progressively synthesize photo-realistic person images based on the appearance features of the source image, the target pose heatmap and the prior transferred content in edge domain. We compare the proposed framework with several state-of-the-art methods to show its superiority in quantitative and qualitative analysis. Moreover, detailed ablation study results demonstrate the efficacy of our contributions. Codes are publicly available at github.com/rocketappslab/SCA-GAN.
BibTeX
@inproceedings{yu2021spatial,
  author = {Wing Yin Yu and Lai-Man Po and Yuzhi Zhao and Jingjing Xiong and Kin Wai Lau},
  title = {{Spatial Content Alignment For Pose Transfer}},
  booktitle = {ICME},
  year = {2021},
  doi = {10.1109/icme51207.2021.9428146},
}
Figure