Human Synthesis and Scene Compositing

Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Andrei Zanfir, Cristian Sminchisescu

Introduction

Generating realistic synthetic images of humans is a long-standing challenge in computer vision, with applications ranging from virtual try-ons and photo editing to data augmentation for training autonomous systems. Historically, research has diverged into two paths: high-fidelity 3D computer graphics, which offer control but often suffer from the "uncanny valley," and 2D deep learning models, which produce realistic textures but frequently fail to maintain geometric consistency.

The HUSC (HUman Synthesis and Scene Compositing) framework bridges this gap. By combining 3D parametric modeling with deep generative networks, HUSC creates realistic images of people in novel poses and integrates them seamlessly into new 3D environments. The system reasons about 3D geometry, perspective, and appearance statistics to move beyond simple "cut-and-paste" methods, producing synthetic images that respect the physical constraints of the real world.

The HUSC pipeline: from query inputs to final appearance-composited scene.

Methodology

The HUSC framework is built upon three primary pillars designed to handle pose synthesis, spatial placement, and visual blending.

1. Human Synthesis (HUSC-S)

This module utilizes a conditional Generative Adversarial Network (GAN) to transform a person's pose. Unlike previous methods that rely solely on 2D skeletons, HUSC employs the SMPL 3D body model as a structural anchor.

A key innovation is the Displacement Field Estimator. This component calculates how pixels should transition from the source image to the target pose. Because standard 3D meshes cannot capture non-rigid elements like flowing hair or loose clothing, the network learns a "delta" ($\Delta M$) to correct the 3D-based displacement field. This allows the system to maintain the high-fidelity texture of the original photo while adapting to the complex physics of fabric movement.

Detailed architecture of the HUSC-S module, featuring the Displacement Field Estimator (A) and the Appearance/Topology encoders (B, C, D).

2. Geometrical Compositing

To place a synthesized person into a new scene, HUSC first estimates the ground plane of the background using depth data and semantic labels. By sampling valid 3D locations on the floor, the system ensures the person is rendered at the correct scale relative to the camera perspective. This geometric reasoning prevents common artifacts like "floating" characters or clipping through environmental objects like furniture.

3. Appearance Compositing

Even a geometrically accurate insertion can appear artificial if the lighting is inconsistent. HUSC employs a refinement network trained to adjust color, contrast, and boundaries. This ensures the synthesized person blends naturally into the ambient light and statistical distribution of the background scene.

Datasets and Evaluation

The researchers utilized several key datasets to train and validate the HUSC framework:

  • DeepFashion: Used for the human synthesis module, providing over 52,000 images of people in various outfits.
  • COCO (Common Objects in Context): Used to train the appearance compositing network to learn natural blending techniques.
  • NYU Depth Dataset V2: Provided the background scenes, RGB images, and aligned depth maps necessary for 3D placement and ground plane estimation.

Key Insights and Results

The HUSC framework achieved state-of-the-art performance on the DeepFashion dataset, reaching an Inception Score (IS) of 3.695 and a Structural Similarity Index (SSIM) of 0.8135, outperforming previous leading models such as DSCF and PATN.

Ablation studies confirmed that the learned displacement field was the primary driver of quality; removing this component caused the IS to drop significantly from 3.69 to 3.38. Furthermore, qualitative results demonstrated the system's ability to handle complex occlusions—such as placing a person behind a desk—by leveraging the scene's depth map, a task that remains difficult for standard 2D synthesis methods.

Limitations and Future Directions

Despite its advancements, HUSC has limitations. The geometrical compositing module currently depends on high-quality depth maps, which are not always available for standard 2D photographs. Additionally, while the appearance module adjusts global color statistics, it does not yet explicitly model complex light interactions like dynamic shadows or reflections on shiny surfaces. Future work may focus on providing more interactive "artistic" control over specific details, such as individual clothing folds or hair styles, while reducing the dependency on pre-existing depth data.

Summary updated

Abstract

Generating good quality and geometrically plausible synthetic images of humans with the ability to control appearance, pose and shape parameters, has become increasingly important for a variety of tasks ranging from photo editing, fashion virtual try-on, to special effects and image compression. In this paper, we propose a HUSC (HUman Synthesis and Scene Compositing) framework for the realistic synthesis of humans with different appearance, in novel poses and scenes. Central to our formulation is 3d reasoning for both people and scenes, in order to produce realistic collages, by correctly modeling perspective effects and occlusion, by taking into account scene semantics and by adequately handling relative scales. Conceptually our framework consists of three components: (1) a human image synthesis model with controllable pose and appearance, based on a parametric representation, (2) a person insertion procedure that leverages the geometry and semantics of the 3d scene, and (3) an appearance compositing process to create a seamless blending between the colors of the scene and the generated human image, and avoid visual artifacts. The performance of our framework is supported by both qualitative and quantitative results, in particular state-of-the art synthesis scores for the DeepFashion dataset.
BibTeX
@inproceedings{zanfir2020human,
  author = {Mihai Zanfir and Elisabeta Oneata and Alin-Ionut Popa and Andrei Zanfir and Cristian Sminchisescu},
  title = {{Human Synthesis and Scene Compositing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6969},
}
Figure