Abstract
Generating good quality and geometrically plausible synthetic images of humans with the ability to control appearance, pose and shape parameters, has become increasingly important for a variety of tasks ranging from photo editing, fashion virtual try-on, to special effects and image compression. In this paper, we propose HUSC, a HUman Synthesis and Scene Compositing framework for the realistic synthesis of humans with different appearance, in novel poses and scenes. Central to our formulation is 3d reasoning for both people and scenes, in order to produce realistic collages, by correctly modeling perspective effects and occlusion, by taking into account scene semantics and by adequately handling relative scales. Conceptually our framework consists of three components: (1) a human image synthesis model with controllable pose and appearance, based on a parametric representation, (2) a person insertion procedure that leverages the geometry and semantics of the 3d scene, and (3) an appearance compositing process to create a seamless blending between the colors of the scene and the generated human image, and avoid visual artifacts. The performance of our framework is supported by both qualitative and quantitative results, in particular state-of-the art synthesis scores for the DeepFashion dataset.
Introduction
Creating realistic images of people in new poses and environments is a cornerstone of applications like virtual try-on, photo editing, and data augmentation for computer vision. Historically, researchers have faced a trade-off between 3D graphics pipelines and 2D generative models. While 3D graphics offer high control and geometric accuracy, they require expensive, manual labor that doesn't scale. Conversely, 2D generative models (GANs) are computationally efficient and produce realistic textures but often fail at geometric consistency—people might appear to float, penetrate objects, or have warped anatomy because the model lacks an inherent understanding of 3D space.
The HUSC framework addresses this by using 3D body models as a structural backbone for a deep generative network, ensuring that synthesized humans are both photorealistic and physically plausible within a 3D scene. By bridging the gap between manual 3D graphics and uncontrollable 2D image generation, HUSC uses 3D geometric reasoning to realistically synthesize humans in novel poses and seamlessly integrate them into complex scenes.
The HUSC Framework
The HUSC framework consists of three main stages: human synthesis, geometrical compositing, and appearance compositing. This end-to-end pipeline allows for the transformation of a foreground query and a pose query into a final, harmonized scene.
Human Synthesis (HUSC-S)
Given a source image and a target 3D pose (represented by the SMPL model), HUSC-S generates the person in the new pose. The core innovation is the Displacement Field Estimator. Instead of relying solely on the 3D model's rigid movement, the network learns a dense displacement field ($M = \Delta M + M_B$). Here, $M_B$ represents the base movement derived from the 3D model, while $\Delta M$ is a learned correction that accounts for non-rigid elements like flowing hair or loose clothing.
The inclusion of a "Topology Encoder" helps the decoder understand the underlying body-part relationships, leading to sharper results in complex poses where limbs might overlap. This combination of structural 3D priors and flexible deep learning allows the model to handle "messy" real-world geometry that pure 3D models often struggle to represent.
Geometrical Compositing
To place the synthesized person in a scene, the system estimates the ground plane using RANSAC on semantic floor points from a depth map. It then samples valid 3D locations, ensuring the person is perpendicular to the floor and correctly scaled. Crucially, it uses 3D collision detection to handle occlusions, such as a person standing behind a desk, ensuring physical plausibility.
Appearance Compositing
To avoid a "cut-and-paste" appearance, an encoder-decoder network harmonizes the foreground person with the background scene. This module adjusts color, lighting, and boundaries to match the scene's statistics while preserving the person's identity.
Experimental Results
The HUSC framework was evaluated using several key datasets, including DeepFashion for human synthesis, Microsoft COCO for appearance harmonization, and the NYU Depth Dataset V2 for diverse indoor background scenes.
On the DeepFashion dataset, HUSC achieved state-of-the-art performance with an Inception Score (IS) of 3.6950 and a Structural Similarity Index (SSIM) of 0.8135, outperforming previous methods like DSCF and DPT. Ablation studies demonstrated that the learned displacement field correction ($\Delta M$) provided the largest performance jump, proving that rigid 3D warping alone is insufficient for high-quality synthesis.
Conclusion
HUSC successfully demonstrates that realistic human synthesis must consider 3D properties like camera viewpoint and floor alignment. While the framework produces high-quality results, limitations remain, including sensitivity to initial 3D body model fitting errors and difficulties generating details not present in the source image (e.g., shoes in a waist-up shot). Future work will explore more sophisticated symmetry constraints and tighter integration of lighting estimation for enhanced realism.
Summary updated