ObjectStitch: Object Compositing with Diffusion Model

Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian L. Price, Jianming Zhang, Soo Ye Kim, Daniel G. Aliaga

Abstract

ObjectStitch is a self-supervised framework that utilizes conditional diffusion models to address the multifaceted challenges of image compositing. By unifying color harmonization, shadow generation, and perspective correction into a single generative task, ObjectStitch produces realistic results without the need for expensive, manually labeled training data. The system employs a specialized Content Adaptor to translate image features into a format compatible with pretrained diffusion models and uses a novel perturbation-based training strategy to learn the physics of light and geometry.

Introduction

Image compositing—the process of inserting an object from one image into another—is a fundamental yet challenging task in digital photography. To achieve a seamless result, several complex variables must be aligned: the object's geometry must match the scene's perspective, its color palette must harmonize with the background lighting, and it must cast realistic shadows.

Most existing solutions tackle these sub-tasks independently, which often results in visual inconsistencies. Furthermore, training these models typically requires large datasets of professional "before and after" composites. ObjectStitch overcomes these hurdles by leveraging the generative power of Stable Diffusion in a unified, self-supervised framework.

Methods

The ObjectStitch architecture adapts a pretrained text-to-image diffusion model for image-based guidance. This is achieved through two primary innovations: the Content Adaptor and a self-supervised training pipeline.

The Content Adaptor

Standard diffusion models are conditioned on text prompts. To enable precise image-based control, the authors developed a Content Adaptor. This component takes features from a pre-trained CLIP image encoder (ViT-L/14) and transforms them into a sequence of visual tokens. This allows the model to preserve both the high-level semantics and the intricate low-level details of the foreground object.

The ObjectStitch pipeline uses a Content Adaptor to condition a pretrained diffusion model on image features.

Self-Supervised Training

To eliminate the need for manual labels, ObjectStitch uses a synthetic data generation technique. Researchers identify an object within a single image and "perturb" it using random rotations, perspective warps, and color shifts. The model is then trained to "reconstruct" the original object within its original context. This process forces the model to learn how to naturally correct geometry and lighting based on the surrounding environment.

Key Insights

  • Unified Generative Modeling: By treating compositing as a denoising problem, the model inherently handles harmonization, shadows, and perspective simultaneously rather than as discrete steps.
  • Image-Based Guidance: The Content Adaptor is crucial because text descriptions are often too vague to capture the specific identity and geometry of a unique object.
  • Learning from Perturbation: Intentionally "breaking" the object's appearance during training provides the necessary signal for the model to learn the underlying physical properties of the scene.

Key Results

ObjectStitch demonstrates superior performance both qualitatively and quantitatively. In a side-by-side user study, ObjectStitch was preferred over traditional copy-paste methods 85% of the time. Quantitatively, the model achieved a Fréchet Inception Distance (FID) of 15.43, significantly lower than the 18.82 achieved by the nearest baseline, indicating higher synthesis quality.

Comparison of ObjectStitch (ours) against copy-paste and traditional baselines across various scenes.

As shown in the comparisons, ObjectStitch excels at adjusting the viewpoint of objects and generating context-aware shadows, such as the shadow cast by the umbrella on the grass or the car on the road.

Limitations and Future Work

Despite its success, ObjectStitch has limitations. It occasionally "hallucinates" changes to an object's identity (e.g., altering a mug's handle) to make it fit the scene. Additionally, because it operates within a bounding box mask, it struggles with long shadows that extend beyond the initial area. Future research may focus on global scene interactions, such as distant reflections, and automated instance mask prediction.

Summary updated

Abstract

Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results. Furthermore, annotating training data pairs for compositing requires substantial manual effort from professionals, and is hardly scalable. Thus, with the recent advances in generative models, in this work, we propose a selfsupervised framework for object compositing by leveraging the power of conditional diffusion models. Our framework can hollistically address the object compositing task in a unified model, transforming the viewpoint, geometry, color and shadow of the generated object while requiring no manual labeling. To preserve the input object's characteristics, we introduce a content adaptor that helps to maintain categori-cal semantics and object appearance. A data augmentation method is further adopted to improve the fidelity of the generator. Our method outperforms relevant baselines in both realism and faithfulness of the synthesized result images in a user study on various real-world images.
BibTeX
@inproceedings{song2023objectstitch,
  author = {Yizhi Song and Zhifei Zhang and Zhe Lin and Scott Cohen and Brian L. Price and Jianming Zhang and Soo Ye Kim and Daniel G. Aliaga},
  title = {{ObjectStitch: Object Compositing with Diffusion Model}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01756},
}
Figure