Abstract
Despite significant progress, controlled generation of complex images with interacting people remains difficult. Existing layout generation methods fall short of synthesizing realistic person instances; while pose-guided generation approaches focus on a single person and assume simple or known backgrounds. To tackle these limitations, we propose a new problem, Persons in Context Synthesis, which aims to synthesize diverse person instance(s) in consistent contexts, with user control over both. The context is specified by the bounding box object layout which lacks shape information, while pose of the person(s) by keypoints which are sparsely annotated. To handle the stark difference in input structures, we proposed two separate neural branches to attentively composite the respective (context/person) inputs into shared "compositional structural space", which encodes shape, location and appearance information for both context and person structures in a disentangled manner. This structural space is then decoded to the image space using multi-level feature modulation strategy, and learned in a self supervised manner from image collections and their corresponding inputs. Extensive experiments on two large-scale datasets (COCO-Stuff and Visual Genome) demonstrate that our framework outperforms state-of-the-art methods w.r.t. synthesis quality.
1. Introduction
Generating realistic images from user-provided layouts is a core challenge in computer vision. Current methods generally fall into two camps: Layout-to-Image models, which use bounding boxes to define object placement, and Pose-Guided Synthesis, which uses keypoints to generate detailed human figures. However, layout-to-image models often produce "blob-like" humans due to the lack of structural information in a bounding box, while pose-guided models typically struggle with complex backgrounds or multiple interacting individuals.
We address this "modality gap" by proposing a framework that integrates coarse spatial constraints (bounding boxes) with sparse structural constraints (keypoints) into a single, coherent scene.
2. Methodology: Compositional Structural Space
The core of our approach is a dual-branch architecture that projects disparate inputs into a shared Compositional Structural Space.
- Context Branch: Processes object labels and style embeddings for background elements using an Instance Sparse Attention Net to create precise masks.
- Person Branch: Takes human keypoints and style embeddings, passing them through a structure generator to create detailed person features.
Instead of generating pixels directly, both branches project their features into a shared 3D structural tensor. This space encodes shape, location, and appearance in a disentangled manner. By blending features in this structural space before final decoding, the model ensures that people and their contexts are visually compatible, handling complex interactions and lighting naturally.
3. Multi-Level Feature Modulation
To transform the structural space into a final image, we employ a decoding strategy where the structural canvas modulates the normalization layers of the generator at multiple scales. This multi-level modulation ensures that the fine-grained details of the structural space are preserved throughout the generation process. Additionally, the use of a Variational Autoencoder (VAE) allows users to sample diverse appearances (textures and colors) for any object or person while keeping their pose and position fixed.
4. Experimental Results
We evaluated our framework on the COCO-Stuff and Visual Genome datasets, creating a specific "Person Split" to measure performance on scenes containing humans.
Our model significantly outperforms baselines like LostGAN and Scene Generation (SG). On COCO-Stuff (256x256), our framework achieved a Frechet Inception Distance (FID) of 76.10, a substantial improvement over SG's 103.80. Human evaluation further validated these results, with volunteers preferring our synthesized persons 75% of the time compared to competing models.
5. Conclusion
By disentangling pose, location, and appearance within a shared structural space, our method enables the high-fidelity synthesis of people in complex contexts. This approach overcomes the limitations of previous layout-to-image and pose-guided models, providing a robust solution for controlled image generation. Future work will focus on extending this structural consistency to video synthesis tasks.
Summary updated