To synthesize images with preferred objects and interactions, a controllable approach involves generating images from a scene graph and a large pool of object crops. In this paradigm, the spatial arrangements of objects are defined by the scene graph, while their specific appearances are determined by retrieved crops. This paper introduces a novel semi-parametric framework that addresses the critical challenges of mutual compatibility between retrieved components and the seamless blending of these components into a final, high-fidelity image.
The Challenge of Mutual Compatibility
Generating realistic images from structured descriptions (scene graphs) is a foundational challenge in computer vision. Existing models generally fall into two categories:
- Parametric models (e.g., standard GANs) often struggle to render fine details or complex interactions from scratch.
- Semi-parametric models attempt to solve this by retrieving real image patches (crops) from a database and "pasting" them together.
However, traditional retrieval methods often select crops independently. This frequently results in "Frankenstein" images where, for example, a sheep crop from a sunny day is paired with a grass crop from a cloudy day, or where the scales and perspectives of retrieved objects are jarringly inconsistent. This research specifically targets the mutual compatibility problem in retrieval and the seamless blending problem in synthesis.
Proposed Methodology
The framework consists of two primary modules designed to ensure both logical consistency and visual harmony.
1. Sequential Crop Selection Module (SCSM)
Instead of retrieving all objects simultaneously, the SCSM treats retrieval as a sequential process. By utilizing a Transformer architecture, the model selects each subsequent crop based on the content, style, and spatial location of all previously chosen crops.
To train this module, the authors employ contrastive learning, allowing the model to distinguish between compatible crops (those originating from the same source image) and incompatible ones. Furthermore, the model explicitly embeds bounding box coordinates into the Transformer, ensuring that retrieved patches are appropriate for their intended spatial positions.
2. Progressive Scene Graph to Image Module (PSGIM)
Once the compatible crops are selected, they must be synthesized into a coherent scene. This is achieved through several sub-components:
- Style Unifier: Calculates an average style across all selected crops to enforce a unified lighting and color scheme.
- Hierarchical Gated Convolutions: The generator operates across multiple scales, using gated convolutions to "paint" the background and fill gaps not covered by retrieved crops.
- Patch-guided Spatially Adaptive Normalization (PSANM): This module uses a grayscale "reference map" of the selected crops to guide the generator. By focusing on structural consistency in grayscale, the generator avoids being confused by minor color mismatches, while the Style Unifier handles the final colorization.
Experimental Results
The model was evaluated on the COCO-Stuff and Visual Genome datasets, both known for their high complexity and dense annotations. The proposed method achieved significant quantitative gains over previous state-of-the-art (SOTA) models such as RetrieveGAN and PasteGAN.
- FID (Fréchet Inception Distance): On COCO-Stuff, the model achieved an FID of 51.6, a notable improvement over RetrieveGAN’s 56.9.
- Inception Score (IS): The model reached 15.2 on COCO-Stuff, compared to 10.2 for the nearest competitor.
- User Preference: In a blind study, participants preferred images generated by this method over 70% of the time when compared to baselines.
Key Insights and Future Directions
A primary takeaway from this work is that compatibility is sequential. Crop selection should not be a search for the "best" individual patch, but a search for the patch that fits best with the current set of selections. Additionally, the use of spatial encoding in the retrieval stage proves that an object's location is as vital as its category when seeking compatible neighbors.
While the model sets a new SOTA, limitations remain. The sequential nature of the Transformer means inference time grows linearly with the number of objects. Furthermore, the entire aesthetic of the generated image is highly dependent on the initial "seed" crop. Future work may explore more efficient search algorithms to scale this approach to massive, web-scale memory banks.
Summary updated