TL;DR
Paste2Place is a physics-guided, one-step model that transforms simple "pasted" images into realistic composites by jointly modeling object relighting and scene effects like shadows and reflections. By using a direct latent bridge and intrinsic physical decomposition, it achieves state-of-the-art quality with sub-second inference speeds, preserving fine details that traditional generative methods often destroy.
What Problem Does This Paper Address?
Image composition—the art of placing a foreground object into a new background—is a staple of visual effects and advertising. However, making a pasted object look "at home" is notoriously difficult.
Current approaches usually fall into two camps:
- Harmonization/Shadow Synthesis: These methods often address only one piece of the puzzle (e.g., just fixing colors or just adding a shadow) and struggle to coordinate the two.
- Generative Insertion: While these produce visually plausible results, they often "hallucinate" or alter the object's identity, blurring fine textures or changing text on labels because they typically start from random noise.
Furthermore, most high-quality generative models require iterative denoising, making them too slow for real-time applications. The authors of this paper sought a unified, efficient solution that preserves the object's identity while perfectly matching the scene's physics.
Methods
The core of Paste2Place is a Source-Anchored Latent Bridge. Instead of starting from pure noise and "dreaming up" an image, the model starts with the latent representation of the pasted input. It then learns a direct, one-step mapping (a "bridge") to the final realistic composite.
To guide this transformation, the authors incorporate a Physics-Guided Framework based on the intrinsic image model: $I = A \odot S + R$ (where $I$ is radiance, $A$ is albedo, $S$ is shading, and $R$ is residual appearance).
The model features:
- Physical Predictor: Estimates changes in albedo, shading, and residuals for both the object and the environment.
- Physical Composer: Analytically recomposes these predictions into "corrections."
- Control Mechanism: These physical corrections are injected into a pretrained Diffusion Transformer (FLUX.2) to guide the image update in a single forward pass.
Datasets
Training such a model requires massive amounts of "before and after" data where the only difference is the physical interaction between the object and the scene. The authors introduced:
- P2P-Data: A massive training set of 86,273 pairs.
- Synthetic (77K): Created using a custom Blender pipeline that provides ground-truth physical annotations (shading, albedo, etc.).
- Real-Scene (9K): Derived from object-removal datasets by perturbing foregrounds and learning to "restore" them to their original state.
- P2P-Bench: A 1,500-example benchmark for evaluation, including a "Wild" subset of manually pasted images to test real-world performance.
Key Insights and Ideas
- Physics as Guidance, Not Just Output: Most models treat physical maps (like depth or shadows) as auxiliary outputs. Paste2Place uses them as internal "instructions" to tell the generator exactly where to darken for shadows or brighten for reflections.
- Identity via Latent Transport: By formulating the task as a "bridge" from the source image rather than a generation from noise, the model naturally retains fine details like the text on a soda can label—a common failure point for models like Stable Diffusion or OmniPaint.
- Unified Interaction: The model recognizes that relighting the object and casting a shadow on the floor are two sides of the same coin (light-scene interaction) and handles them simultaneously.
Key Results
Paste2Place outperforms 11 state-of-the-art methods across both quantitative metrics and human preference studies.
- Quality: It improved PSNR by 6.48 dB over the strongest baseline (MV-CoLight) and reduced perceptual error (LPIPS) by 40.4% relative to DreamFuse.
- Speed: The model achieves an average end-to-end latency of 0.743 seconds on an NVIDIA H100 GPU. This is significantly faster than iterative methods, which can take anywhere from 8 to 115 seconds.
- Detail Preservation: Qualitative results show that while other models often "re-draw" labels or blur textures, Paste2Place keeps them crisp while perfectly integrating them into the lighting environment.
Limitations and Future Work
While the paper demonstrates impressive results, some limitations remain:
- Synthetic Bias: The model relies heavily on synthetic data for physical supervision. While the real-scene data helps, there is always a gap between rendered physics and complex real-world optics.
- Placement Logic: The model focuses on how an object looks once placed, but it doesn't decide where it should go. Integrating a "placement suggester" could be a logical next step.
- Extreme Cases: The authors noted that human review was still necessary to filter out "visibly implausible" synthetic compositions during dataset creation, suggesting the automated pipeline still has room for improvement in handling complex geometry collisions.
Summary updated