WaveIPT: Joint Attention and Flow Alignment in the Wavelet domain for Pose Transfer

Liyuan Ma, Tingwei Gao, Haitian Jiang, Haibin Shen, Kejie HuangICCV 2023 · DOI

Introduction

Human pose transfer aims to synthesize a realistic image of a person in a target pose while preserving the identity and appearance details from a source image. This task is foundational for applications like virtual try-on and digital content creation. However, existing methods often struggle with a fundamental trade-off: global structural consistency versus local textural detail.

Traditional approaches typically rely on either attention mechanisms or flow-based alignment. Attention mechanisms are adept at capturing global semantic structures (e.g., general body shape) but often produce blurry textures. Conversely, flow-based alignment warps pixels to preserve fine details (e.g., clothing patterns) but is prone to artifacts and geometric distortions during large pose changes. Fusing these two paradigms in the standard spatial domain often leads to mutual interference rather than synergy.

Frequency-Aware Alignment with WaveIPT

To address these limitations, the authors propose WaveIPT (Wavelet-aware Image-based Pose Transfer). The core innovation lies in performing feature alignment and fusion in the frequency domain using the Discrete Wavelet Transform (DWT). By decomposing features into Low Frequency (LF) and High Frequency (HF) components, the model can leverage the specific strengths of attention and flow where they are most effective.

Motivation for WaveIPT: Attention mechanisms excel at capturing low-frequency (LF) semantic structures, while flow-based alignment is superior for preserving high-frequency (HF) textures. WaveIPT fuses these in the wavelet domain to achieve superior results.

As illustrated in the figure above, attention alignment naturally captures the coarse semantic layout (LF-pass), while flow alignment excels at preserving sharp edges and textures (HF-pass). WaveIPT formalizes this observation by processing these components separately and then fusing them adaptively.

Methodology

The WaveIPT architecture consists of several key modules designed to handle multi-scale frequency interactions:

  1. Intra-scale Local Correlation (ILC): This module adaptively fuses attention and flow features within the same frequency scale. It uses local correlation to ensure that texture alignment is guided by relevant local context, preventing distant, irrelevant features from introducing noise.
  2. Inter-scale Feature Interaction (IFI): To ensure consistency across the frequency spectrum, the IFI module facilitates communication between LF and HF scales. This allows the coarse structural information from the LF components to guide the refinement of high-frequency details.
  3. Progressive Flow Regularization (PFR): To handle extreme pose variations, WaveIPT introduces an intermediate "bridge" pose during training. By decomposing a large transformation into smaller steps (Source → Intermediate → Target), the model learns more stable and accurate flow fields, reducing geometric artifacts.
The WaveIPT architecture. Features from the source image ($I_s$) and target pose ($P_t$) are encoded and decomposed via DWT. The features then undergo Intra-scale Local Correlation and Inter-scale Feature Interaction before being reconstructed via Inverse DWT (IDWT).

Experimental Results

WaveIPT was evaluated on the DeepFashion benchmark at both standard (256 × 176) and high resolutions (512 × 352). The model achieved a new state-of-the-art, outperforming previous methods like NTED and FreqHPT.

  • Quantitative Performance: WaveIPT demonstrated significant improvements in realism and perceptual similarity, reducing the FID (Fréchet Inception Distance) by 4.97% and the LPIPS (Learned Perceptual Image Patch Similarity) by 3.89%.
  • Qualitative Quality: Visual comparisons show that WaveIPT superiorly preserves complex clothing logos, fabric textures, and sharp limb boundaries. It effectively avoids the "melting" or blurring effects common in previous attention-only or flow-only architectures.
  • Human Evaluation: In user studies, participants consistently preferred WaveIPT's outputs for their superior realism and consistency with the source identity.

Conclusion

WaveIPT demonstrates that the frequency domain is a powerful space for fusing complementary alignment strategies. By separating semantic structure from textural detail via wavelets, the framework overcomes the traditional limitations of spatial-domain fusion. Future work may explore extending this wavelet-aware approach to broader tasks such as general image inpainting and virtual try-on systems.

Summary updated

BibTeX
@inproceedings{ma2023waveipt,
  author = {Liyuan Ma and Tingwei Gao and Haitian Jiang and Haibin Shen and Kejie Huang},
  title = {{WaveIPT: Joint Attention and Flow Alignment in the Wavelet domain for Pose Transfer}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00663},
}
Figure