TL;DR
This paper introduces a novel semi-supervised framework for synthesizing realistic person images with arbitrary poses, shapes, and views. By leveraging SMPL model-derived UV maps and a cross-training approach, the method effectively disentangles human appearance from pose and shape, enabling the generation of consistent and high-quality images even for unseen viewpoints.
What Problem Does This Paper Address?
Synthesizing realistic human images with control over pose, shape, and view has been a long-standing challenge. Existing methods often rely on 2D keypoints, which lack crucial shape information and can be inaccurate due to occlusions, leading to mismatches between appearance and structure. While some approaches attempt to disentangle these factors, they struggle with maintaining a consistent global appearance, especially when generating images from unseen viewpoints. This can result in inaccurate or blurry renderings. The paper aims to address this by developing a method that can generate diverse person images while preserving a consistent, global appearance of the target individual across various poses, shapes, and camera angles.
Methods
The proposed approach is a two-stage, semi-supervised framework.
Stage 1: Training a Generalizable Network 1. SMPL Model and UV Maps: The authors utilize the SMPL (Skinned Multi-Person Linear Model) to obtain 3D human body parameters (pose, shape, camera). These parameters are then used to render UV maps, which provide a structured representation of the human body's surface. 2. UV-Guided Appearance Encoder: A generative adversarial network (GAN) is trained with a UV-guided appearance encoder. This encoder takes an input image and its corresponding UV map to extract a latent code representing the person's appearance. The UV maps act as a guide, helping the encoder learn appearance information that is aligned with the human structure. 3. Cross-Training for Disentanglement: To further separate appearance from pose and shape, a cross-training strategy is employed. Two images of the same person are processed, their UV maps are exchanged, and then fed into the decoder. This forces the appearance encoder to learn features that are invariant to pose changes. 4. Decoder and GAN Training: A decoder reconstructs the image from the appearance latent code and UV maps. The entire network is trained adversarially to produce realistic images.
Stage 2: Fine-tuning for Global Appearance Enrollment 1. Person-Specific Fine-tuning: After the initial training, the network is fine-tuned using a set of images of a specific target person. This fine-tuning process enrolls the person's global appearance into the network. 2. Synthesis: At test time, the fine-tuned network can generate images of the target person in new poses, shapes, and views by varying the SMPL parameters.
Datasets
- Human3.6M Dataset: Used for pre-training the network to generalize across various poses. It contains data from multiple subjects performing different activities.
- Self-Collected Dataset: A dataset of videos captured by mobile phones of specific individuals. This dataset is used for fine-tuning and demonstrating the person-specific appearance enrollment.
Key Insights and Ideas
- UV Maps as Structural Guidance: Using UV maps rendered from the SMPL model is a more effective way to represent human structure than traditional 2D keypoints, as they encode shape information and are more robust to occlusion.
- Cross-Training for Disentanglement: The cross-exchange of UV maps between images during training is a clever technique to enforce the separation of appearance from pose and shape, leading to more disentangled representations.
- Semi-Supervised Approach: The reliance on a pretrained HMR model and the use of unlabeled data for the initial training stage make the approach semi-supervised, reducing the need for extensive manual annotation.
- Global Appearance Enrollment: Fine-tuning with a specific person's images is crucial for capturing their unique global appearance, enabling consistent synthesis across different conditions.
Key Results
- Qualitative Results: The method successfully generates realistic person images with arbitrary poses, shapes, and views. Qualitative comparisons show improvements over previous methods, particularly in maintaining sharp details and accurate body structure.
- Quantitative Results: The paper reports improved Inception Scores (IS) and Structural Similarities (SSIM) on the Human3.6M dataset compared to a previous work (Dance [3]), indicating better image quality and realism.
- Shape Control: The framework allows for easy manipulation of human shape (weight and height) by adjusting SMPL parameters, demonstrating flexibility.
- Novel View Synthesis: The fine-tuned model can synthesize images from unseen views with accurate appearance, showcasing the effectiveness of global appearance enrollment.
- Enrollment Amount Study: The results indicate that around 500 images are sufficient for effective appearance enrollment, with diminishing returns beyond that point.
Limitations and Future Work
- Limitations: The paper acknowledges that while the network is trained on a specific person for enrollment, it requires a set of images of that person. The quality of the generated images is dependent on the quality and diversity of the enrollment data. The current method focuses on human body synthesis and does not explicitly handle face generation.
- Future Work: The authors suggest extending the method to include face generation for more complete and visually appealing results. They also aim to improve the method for generating clearer images and explore its application in more diverse fields.
Summary updated