Person-in-Context Synthesis with Compositional Structural Space

Weidong Yin, Ziwei Liu, Leonid Sigal

TL;DR

This paper introduces Person-in-Context Synthesis, a new generative task that creates realistic images of multiple people interacting within complex scenes. By projecting coarse bounding box layouts and sparse human keypoints into a shared "compositional structural space," the authors successfully disentangle pose, location, and appearance to generate high-fidelity images that outperform existing state-of-the-art layout-to-image models.

What Problem Does This Paper Address?

Generating realistic images from user-provided layouts is a "holy grail" of computer vision, but current methods hit a wall when it comes to humans.

Existing approaches generally fall into two camps:

  1. Layout-to-Image: These use bounding boxes to define where objects should be. While they handle backgrounds well, they usually produce "blob-like" or distorted humans because a simple box doesn't provide enough information about human articulation.
  2. Pose-Guided Synthesis: These use keypoints to generate highly detailed people but usually assume a simple, static, or pre-known background. They struggle to model multiple people or complex environmental interactions.

The challenge lies in the modality gap: bounding boxes are coarse spatial constraints, while keypoints are sparse structural constraints. Combining them into a single, coherent scene without visible "seams" or artifacts is incredibly difficult.

Methods

The authors propose a dual-branch framework that merges these different inputs into a unified intermediate representation.

  • The Two-Branch Architecture:
    • Context Branch: Processes object labels and style embeddings for background elements (trees, buildings, etc.) using an Instance Sparse Attention Net to create precise masks.
  • Person Branch: Takes human keypoints and style embeddings, passing them through a structure generator to create detailed "person features."
  • Compositional Structural Space: This is the core innovation. Instead of generating pixels directly, both branches project their features into a shared 3D structural tensor. This space encodes shape, location, and appearance in a disentangled way. Because it's a shared space, person and context features can be combined via simple linear summation.
  • Diverse Style Embeddings: The model uses a Variational Autoencoder (VAE) to learn a latent space of appearances. At test time, users can sample from a Gaussian distribution to change the "look" (texture/color) of an object or person without changing their pose or position.
  • Multi-Level Feature Modulation: To turn the structural space into a final image, the model uses a decoding strategy similar to SPADE (GauGAN), where the structural canvas modulates the normalization layers of the generator at multiple scales.

Datasets

The model was evaluated on two massive benchmarks:

  • COCO-Stuff (2017): A standard for scene generation containing 182 categories.
  • Visual Genome (v1.4): Used for its dense annotations of objects and relationships.
  • The "Person Split": Since the paper focuses on humans, the authors created a specific evaluation subset from these datasets containing only images with at least one person.
  • "Person Crop": A secondary dataset of 64x64 human patches used to specifically measure the quality of synthesized people compared to specialized pose-to-person models.

Key Insights and Ideas

  • Structural Blending > Image Blending: The authors found that trying to "paste" a generated person onto a generated background in the final image space leads to unnatural boundaries. By blending them in the structural space before the final decoding, the model learns to make the person and context visually compatible (e.g., correct lighting and contact points).
  • Disentanglement is Key: Because the structural space separates pose from style, the model can perform "Person Reenactment." You can keep a person's appearance (style code) constant while interpolating their keypoints to make them move or change posture within the synthesized scene.
  • Modality Independence: Interestingly, the authors found that using more "powerful" annotations like DensePose (which provides a full 3D surface map) didn't significantly outperform their keypoint-based method. This suggests that the way features are composed is more important than the density of the input annotation.

Key Results

The proposed framework significantly outperformed baselines like LostGAN and Scene Generation (SG) across several metrics:

  • FID (Frechet Inception Distance): On COCO-Stuff (256x256), the model achieved an FID of 76.10, compared to SG's 103.80. Lower FID indicates images that are statistically closer to real photos.
  • Classification Accuracy: A pre-trained classifier was more likely to correctly identify the objects and people in the authors' generated images than in those of the competitors, proving higher recognizable detail.
  • User Study: In a head-to-head comparison, human volunteers preferred the visual quality of the authors' synthesized persons 75% of the time, compared to just 15% for LostGAN.
  • Complexity Scaling: Unlike previous models, the authors' method maintains high quality even as the number of people in the scene increases.

Limitations and Future Work

  • Diversity Constraints: While the model can sample different textures, the "Diversity Score" was lower than some baselines. This is because the model adheres very strictly to the structural space; it won't change the underlying shape or pose of an object during random sampling, only its appearance.
  • Keypoint Dependency: The quality of the output is heavily dependent on the quality of the input keypoints. For datasets like Visual Genome, the authors had to rely on an automated detector (AlphaPose), which can introduce its own errors.
  • Future Directions: The authors suggest that extending this to video synthesis—where temporal consistency in the structural space would be vital—is a natural next step.

Summary updated

Abstract

Despite significant progress, controlled generation of complex images with interacting people remains difficult. Existing layout generation methods fall short of synthesizing realistic person instances; while pose-guided generation approaches focus on a single person and assume simple or known backgrounds. To tackle these limitations, we propose a new problem, Persons in Context Synthesis, which aims to synthesize diverse person instance(s) in consistent contexts, with user control over both. The context is specified by the bounding box object layout which lacks shape information, while pose of the person(s) by keypoints which are sparsely annotated. To handle the stark difference in input structures, we proposed two separate neural branches to attentively composite the respective (context/person) inputs into shared "compositional structural space", which encodes shape, location and appearance information for both context and person structures in a disentangled manner. This structural space is then decoded to the image space using multi-level feature modulation strategy, and learned in a self supervised manner from image collections and their corresponding inputs. Extensive experiments on two large-scale datasets (COCO-Stuff [6] and Visual Genome [19]) demonstrate that our framework outperforms state-of-theart methods w.r.t. synthesis quality.
BibTeX
@inproceedings{yin2021person,
  author = {Weidong Yin and Ziwei Liu and Leonid Sigal},
  title = {{Person-in-Context Synthesis with Compositional Structural Space}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00287},
}
Figure