Abstract
ObjectStitch is a self-supervised framework that utilizes conditional diffusion models to address the multifaceted challenges of image compositing. By unifying color harmonization, shadow generation, and perspective correction into a single generative task, ObjectStitch produces realistic results without the need for expensive, manually labeled training data. The system employs a specialized Content Adaptor to translate image features into a format compatible with pretrained diffusion models and uses a novel perturbation-based training strategy to learn the physics of light and geometry.
Introduction
Image compositing—the process of inserting an object from one image into another—is a fundamental yet challenging task in digital photography. To achieve a seamless result, several complex variables must be aligned: the object's geometry must match the scene's perspective, its color palette must harmonize with the background lighting, and it must cast realistic shadows.
Most existing solutions tackle these sub-tasks independently, which often results in visual inconsistencies. Furthermore, training these models typically requires large datasets of professional "before and after" composites. ObjectStitch overcomes these hurdles by leveraging the generative power of Stable Diffusion in a unified, self-supervised framework.
Methods
The ObjectStitch architecture adapts a pretrained text-to-image diffusion model for image-based guidance. This is achieved through two primary innovations: the Content Adaptor and a self-supervised training pipeline.
The Content Adaptor
Standard diffusion models are conditioned on text prompts. To enable precise image-based control, the authors developed a Content Adaptor. This component takes features from a pre-trained CLIP image encoder (ViT-L/14) and transforms them into a sequence of visual tokens. This allows the model to preserve both the high-level semantics and the intricate low-level details of the foreground object.
Self-Supervised Training
To eliminate the need for manual labels, ObjectStitch uses a synthetic data generation technique. Researchers identify an object within a single image and "perturb" it using random rotations, perspective warps, and color shifts. The model is then trained to "reconstruct" the original object within its original context. This process forces the model to learn how to naturally correct geometry and lighting based on the surrounding environment.
Key Insights
- Unified Generative Modeling: By treating compositing as a denoising problem, the model inherently handles harmonization, shadows, and perspective simultaneously rather than as discrete steps.
- Image-Based Guidance: The Content Adaptor is crucial because text descriptions are often too vague to capture the specific identity and geometry of a unique object.
- Learning from Perturbation: Intentionally "breaking" the object's appearance during training provides the necessary signal for the model to learn the underlying physical properties of the scene.
Key Results
ObjectStitch demonstrates superior performance both qualitatively and quantitatively. In a side-by-side user study, ObjectStitch was preferred over traditional copy-paste methods 85% of the time. Quantitatively, the model achieved a Fréchet Inception Distance (FID) of 15.43, significantly lower than the 18.82 achieved by the nearest baseline, indicating higher synthesis quality.
As shown in the comparisons, ObjectStitch excels at adjusting the viewpoint of objects and generating context-aware shadows, such as the shadow cast by the umbrella on the grass or the car on the road.
Limitations and Future Work
Despite its success, ObjectStitch has limitations. It occasionally "hallucinates" changes to an object's identity (e.g., altering a mug's handle) to make it fit the scene. Additionally, because it operates within a bounding box mask, it struggles with long shadows that extend beyond the initial area. Future research may focus on global scene interactions, such as distant reflections, and automated instance mask prediction.
Summary updated