Spatial Content Alignment for Pose Transfer

Wing Yin Yu, Lai-Man Po, Yuzhi Zhao, Jingjing Xiong, Kin Wai Lau

TL;DR

SCA-GAN introduces a two-phase framework for pose transfer that solves the "spatial misalignment" problem by first predicting a person's edge map in the target pose. By using these predicted edges as a high-frequency spatial guide and employing a novel normalization technique (CS-SPADE), the model generates photo-realistic images with significantly better garment textures and facial details than previous state-of-the-art methods.

What Problem Does This Paper Address?

Pose transfer—the task of synthesizing an image of a person in a new pose based on a source image—is notoriously difficult because the source and target images are "spatially misaligned."

Existing methods generally fall into two camps:

  1. Warping-based: These try to "stretch" the source image to fit the new pose. This often results in blurry boundaries and artifacts because geometric matching is rarely perfect.
  2. Attention-based: These use neural networks to "attend" to relevant parts of the source image. While better at fusing style, they often lack enough "content guidance," leading to distorted clothing textures and loss of facial identity.

The authors identify that the bottleneck is a lack of reliable spatial hints to guide the reconstruction of fine-grained details like garment patterns and human characteristics.

Methods

The authors propose the Spatial Content Alignment GAN (SCA-GAN), which operates in two distinct phases:

Phase 1: Prior Content Transfer Network (PCT-Net)

Instead of jumping straight to pixels, the model first predicts what the person's edges would look like in the target pose.

  • Why Edges? Edges represent high-frequency signals that capture the "skeleton" of both the person's identity and their clothing.
  • The PCT-Net takes the source edge map, source pose, and target pose to generate a "target edge map." This provides a spatially aligned "blueprint" for the next phase.

Phase 2: Image Synthesis Network (IS-Net)

This network takes the source image and the predicted edges from Phase 1 to create the final photo. It features two novel components:

  • Style Encoder Blocks (Style EnBlk): Unlike standard residual blocks, these omit normalization layers. The authors argue that normalization can wash out unique, non-generalizable patterns (like a specific logo or texture). Removing it helps the encoder capture "unique pattern statistics."
  • Content-Style SPADE (CS-SPADE): This is an evolution of the Spatially-Adaptive Denormalization (SPADE) technique. It simultaneously injects aligned content (the target pose and predicted edges) and misaligned style (features from the source image) during the upsampling process.

Datasets

The researchers evaluated their model on the DeepFashion (In-shop Clothes Retrieval Benchmark).

  • Scale: 101,966 training pairs and 8,570 testing pairs.
  • Preprocessing: They used the XDoG (Extended Difference-of-Gaussians) algorithm for edge detection and HPE for pose detection. All images were resized to 256 × 176.

Key Insights and Ideas

  • Edges as a Bridge: The most significant "aha moment" is using pixel-level edge maps as an intermediate representation. Edges are easier to transfer than full RGB pixels but provide much richer spatial guidance than simple pose heatmaps or coarse semantic labels.
  • Normalization is a Double-Edged Sword: While normalization helps GANs converge, the authors found that in the encoding stage, it actually hinders the network's ability to remember specific, high-fidelity textures.
  • Decoupling Content and Style: By predicting the "where" (edges) before the "what" (style/texture), the synthesis network doesn't have to struggle with moving pixels around; it can focus entirely on filling in the details.

Key Results

SCA-GAN outperformed several state-of-the-art models (including PATN, ADGAN, and APS) across multiple metrics:

  • Quantitative: It achieved the highest Inception Score (IS) of 3.497 and the lowest FID (11.676) and LPIPS (0.167), indicating superior image quality and perceptual similarity to real photos.
  • Qualitative: Visual results show much sharper clothing boundaries, realistic creases in dresses, and better preservation of facial features compared to competitors, which often produced "blobby" or distorted results in those areas.
  • Ablation Study: Removing the "prior edge transfer" caused the FID score to jump from 11.6 to 12.4, proving that the edge-prediction step is crucial for performance.

Limitations and Future Work

  • Edge Transfer Failures: The authors note that if the PCT-Net fails to predict edges correctly (due to class imbalances in the dataset or extremely rare poses), the final synthesis will suffer.
  • Complex Backgrounds: While the model excels at person-centric synthesis, the paper focuses on the DeepFashion dataset, which typically has clean, simple backgrounds. Performance on complex, cluttered real-world backgrounds remains an open question.
  • Future Direction: The authors suggest further exploring the "complementary stimulation" between the Style Encoder and the Synthesis Network to improve the fusion of style and content even further.

Summary updated

Abstract

Due to unreliable geometric matching and content misalignment, most conventional pose transfer algorithms fail to generate fine-trained person images. In this paper, we propose a novel framework – Spatial Content Alignment GAN (SCA-GAN) which aims to enhance the content consistency of garment textures and the details of human characteristics. We first alleviate the spatial misalignment by transferring the edge content to the target pose in advance. Secondly, we introduce a new Content-Style DeBlk which can progressively synthesize photo-realistic person images based on the appearance features of the source image, the target pose heatmap and the prior transferred content in edge domain. We compare the proposed framework with several state-of-the-art methods to show its superiority in quantitative and qualitative analysis. Moreover, detailed ablation study results demonstrate the efficacy of our contributions. Codes are publicly available at github.com/rocketappslab/SCA-GAN.
BibTeX
@inproceedings{yu2021spatial,
  author = {Wing Yin Yu and Lai-Man Po and Yuzhi Zhao and Jingjing Xiong and Kin Wai Lau},
  title = {{Spatial Content Alignment for Pose Transfer}},
  booktitle = {ICME},
  year = {2021},
  doi = {10.1109/icme51207.2021.9428146},
}

Loading related content...

Figure