Direct May Not Be the Best: An Incremental Evolution View of Pose Generation

Yuelong Li, Tengfei Xiao, Lei Geng, Jianming WangAAAI 2024 · DOI

Introduction

Human pose generation—the task of synthesizing an image of a person in a target pose given a source image—is a complex challenge in computer vision. The core difficulty stems from the "dimension degrading" nature of 2D images: as a 3D body moves, parts become occluded, and the mapping between source and target pixels becomes highly non-linear.

Most existing state-of-the-art (SOTA) models attempt a "direct" or "one-to-one" transformation, leaping from the source to the final target in a single step. However, this often leads to significant content distortion and blurring because the model struggles to bridge massive visual gaps instantaneously. This paper proposes a paradigm shift: Incremental Evolution, which breaks down dramatic transformations into a series of manageable, fine-grained steps.

Comparison between the traditional one-to-one pose generation approach and the proposed incremental evolution framework.

Methodology

The proposed framework centers on an iterative process guided by three primary components designed to maintain anatomical consistency and texture fidelity.

1. Global Evolution Constraints

Before generating pixels, the model establishes a roadmap using a bidirectional Recurrent Neural Network (RNN). This network generates a sequence of intermediate "pose skeletons" and "semantic maps" (parsing maps). This ensures that the transition from the start pose to the end pose is both anatomically logical and consistent across the entire sequence.

2. Incremental Evolution (IE) Blocks

To capture visual features at different granularities, the model employs IE Blocks featuring multi-scale kernels (3x3, 5x5, and 7x7) and scale-wise attention. These blocks are specifically designed to model the "evolutionary tendency," predicting how features should shift from one incremental step to the next.

3. Triple-Path Knowledge Fusion (TPKF)

The synthesis engine relies on a Triple-Path Knowledge Fusion structure. At each iteration $t$, the model fuses information from three distinct paths to generate the next intermediate state:

  • Path 1 (Incremental): Utilizes incremental evolution constraints (the "tendency" derived from previous steps).
  • Path 2 (Source): Incorporates source image features to act as an "identity" and "texture" anchor.
  • Path 3 (Target): Integrates the current target pose and semantics, defining the "destination" for that specific step.
The Triple-Path Fusion architecture at iteration t, showing how incremental constraints, source features, and target semantics are integrated.

By iterating through these paths, the model gradually "evolves" the source image into the target, using the output of iteration $t$ as feedback to guide iteration $t+1$. This self-correcting mechanism is crucial for preserving complex textures, such as clothing patterns, across large transformations.

Experimental Evaluation

The researchers evaluated their approach on three datasets: a new Turning-Round dataset (designed for incremental transitions), the standard DeepFashion benchmark, and the Tai-Chi motion video dataset.

Key Results

The "Evolution View" consistently outperformed previous SOTA models (including PISE, CASD, and DPTN):

  • Quantitative Performance: On the Fashion dataset, the model achieved an SSIM of 0.930 and a PSNR of 25.332. It also reached an FID of 12.720, indicating significantly higher image realism than its predecessors.
  • Ablation Studies: Results confirmed that increasing the number of evolutionary increments (from zero to five) consistently improved image quality metrics.
  • Human Perception: In a "Real vs. Generated" study, volunteers identified the proposed model's outputs as "real" 20.8% of the time, nearly doubling the performance of older models like PISE (11.2%).

Conclusion

By adopting a "divide and conquer" strategy for non-linear pose transformations, the incremental evolution framework achieves higher fidelity and fewer distortions than direct methods. While the iterative nature of the model increases computational cost (higher MACs), the gain in visual quality and the generation of useful intermediate poses—valuable for tasks like video interpolation—represent a significant advancement in human pose synthesis. Future work will focus on optimizing inference time and extending the framework to multi-person interactions.

Summary updated

BibTeX
@inproceedings{li2024direct,
  author = {Yuelong Li and Tengfei Xiao and Lei Geng and Jianming Wang},
  title = {{Direct May Not Be the Best: An Incremental Evolution View of Pose Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28112},
}
Figure