Abstract
Pose-guided person image generation involves re-rendering a person from a source image into a new target pose while preserving identity and appearance. This paper introduces a novel p-Norm regression (pNR) module that treats this complex task as a linear operation in a latent feature space. By assuming appearance is the product of pose and a pose-invariant identity feature, the proposed framework provides a unified solution for supervised and unsupervised training, as well as single- and multi-shot generation.
Introduction
Generating realistic images of people in novel poses is a significant challenge in computer vision due to the highly deformable nature of the human body. Traditional methods often rely on complex spatial transformations or heavy attention mechanisms. These approaches frequently suffer from two major limitations: a strict dependency on identity labels during training and an inability to easily incorporate multiple source images (multi-shot) to improve generation quality.
Methodology: The p-Norm Regression Module
The core innovation of this work is the p-Norm regression (pNR) module. The authors propose a simple yet powerful linear assumption in the hidden feature space: Appearance ($H$) $\approx$ Pose ($P$) $\times$ Identity ($F$)
In this formulation, $F$ represents a pose-invariant identity matrix. The generation process follows three primary stages:
- Identity Estimation: Given source appearance features ($H_s$) and pose features ($P_s$), the module solves a regression problem to estimate the identity matrix $F$.
- Appearance Prediction: The estimated $F$ is multiplied by target pose features ($P_t$) to predict the target appearance features ($H_t$).
- Image Synthesis: A Generative Adversarial Network (GAN) generator $G$ decodes the predicted features into a final image $I_g$.
The regression can be implemented using different norms:
- LSE ($p=2$): Least-squares error, offering a closed-form solution.
- LAD ($p=1$): Least absolute deviation, which is more robust to outliers such as background noise. This version uses an Iterative Re-weighted Least Squares (IRLS) solver to maintain differentiability for end-to-end training.
This framework is inherently flexible. For unsupervised training, the model functions as a denoising auto-encoder. For multi-shot generation, additional source images are simply appended as extra rows in the regression matrices, allowing for more accurate identity estimation without architectural changes.
Experimental Setup
The model was evaluated on the Market-1501 dataset, a benchmark for person re-identification characterized by 128x64 resolution images with diverse backgrounds and lighting. Poses were represented using 18-joint keypoint heatmaps extracted via a standard Human Pose Estimator (HPE).
Results and Analysis
The pNR module demonstrates significant improvements over existing baselines. In supervised settings, the pNR (LAD) variant achieved an Inception Score (IS) of 3.631, surpassing models like PATN (3.323). Remarkably, the unsupervised version (pNR) achieved an IS of 3.681*, setting a new state-of-the-art for unsupervised pose-guided generation.
Key Insights
- Latent Linearity: The success of the $H \approx PF$ assumption suggests that complex pose-appearance relationships can be linearized in the right hidden space, bypassing the need for computationally expensive spatial warping.
- Robustness of $L_1$: The $p=1$ (LAD) norm consistently outperformed $p=2$ (LSE), indicating that the latent space contains significant "outliers" (e.g., background clutter) that the $L_1$ norm handles more effectively.
- Multi-shot Scalability: Quantitative metrics like SSIM improved as the number of source images ($M$) increased (from 0.305 at $M=1$ to 0.313 at $M=5$), with qualitative results showing sharper clothing textures.
Conclusion and Future Work
While the pNR module offers a flexible and high-performing framework, some limitations remain. Unsupervised results, though numerically strong, can exhibit more visual artifacts than supervised ones. Additionally, the linear assumption may struggle with extreme self-occlusions or lighting changes. Future research will focus on scaling this approach to higher resolutions and exploring non-linear extensions to the identity-pose relationship.
Summary updated