Abstract
Generating photo-realistic images of humans across multiple camera views simultaneously is a significant challenge due to the requirement of strict pose consistency. We propose a framework that supervises a joint learning process of multiple image-to-image translation models using 3D pose constraints. By enforcing consistency in 3D space, our method ensures that synthesized subjects appear in physically plausible and synchronized postures regardless of the viewing angle, effectively addressing the limitations of independent single-view generation models.
Introduction
Human image generation is a complex problem in computer vision because the human body is non-rigid, highly articulated, and capable of a vast range of motion. While "Pose-guided Person Image Generation" has seen success in single-view tasks, such as changing a person's pose or clothing, these methods often struggle with multi-view consistency. When independent models are trained for different camera angles, they frequently produce conflicting 2D poses—for instance, a limb might appear raised in one view but lowered in another. This paper addresses the core challenge of view-consistency by ensuring that the 2D projections in all generated views are mathematically consistent with a single, shared 3D human pose.
Methodology
Our framework is built upon the foundation of CycleGAN, a popular architecture for unpaired image-to-image translation. We extend this by implementing a joint learning architecture where multiple translation modules (one per viewpoint) are trained simultaneously.
The primary innovation is the integration of 3D pose supervision. During the training phase, the generated "fake" images for each view are processed through a pre-trained 3D pose estimation network. A 3D loss term is then calculated based on the Euclidean distance between the predicted 3D pose and the ground-truth 3D pose of the source subject. This constraint forces the generators to produce images where the 2D poses are consistent projections of the same 3D skeleton. Additionally, a pose-scaling function is employed to ensure the comparison remains invariant to individual body proportions, such as varying limb lengths.
Experimental Evaluation
The researchers evaluated the proposed method using the CMU-Panoptic dataset. This specialized dataset features 40 subjects captured by up to 31 synchronized cameras within a "Panoptic Studio" dome. Crucially, it provides high-frequency video and accurate 3D skeleton annotations for every frame, making it an ideal benchmark for testing multi-view consistency and 3D-supervised learning.
Results and Discussion
The proposed method demonstrates clear qualitative and quantitative advantages over standard CycleGAN baselines.
Qualitative Superiority and Pose Stability
In side-by-side comparisons, our method consistently preserves the source person's pose across all views. Standard CycleGANs often suffer from "pose collapse," where the model fails on difficult or unusual postures and instead hallucinates a more common pose. The 3D constraint acts as a stabilizer, preventing this drift and ensuring that limbs are rendered correctly in secondary views. Furthermore, the model showed a "context completion" effect; by anchoring the global 3D pose, the GAN was better able to fill in fine-grained anatomical details like finger orientation correctly.
Temporal Stability and Generalization
Although the model was trained on individual frames without an explicit temporal loss, it demonstrated surprising stability across video sequences. The 3D constraint effectively anchored the subject's movement, leading to smoother transitions compared to the jittery results of the baseline. The model also generalized well across different pairs of subjects and camera angles, maintaining consistency in both "Person A to Person B" and "Person B to Person A" translations.
Conclusion
We have presented a novel framework that utilizes 3D pose as a universal anchor for multi-view image translation. While the results are promising, limitations remain regarding view scalability, as the current framework requires one translation model per camera. Future work will explore more unified architectures to handle arbitrary viewpoints and test performance in complex, "in-the-wild" environments with cluttered backgrounds.
Summary updated