Cut-and-Paste Neural Rendering

Anand Bhattad, David A. Forsyth

Abstract

Cut-and-paste methods take an object from one image and insert it into another. Doing so often results in unrealistic looking images because the inserted object's shading is inconsistent with the target scene's shading. Existing reshading methods require a geometric and physical model of the inserted object, which is then rendered using environment parameters. Accurately constructing such a model only from a single image is beyond the current understanding of computer vision. We describe an alternative procedure -- cut-and-paste neural rendering, to render the inserted fragment's shading field consistent with the target scene. We use a Deep Image Prior (DIP) as a neural renderer trained to render an image with consistent image decomposition inferences. The resulting rendering from DIP should have an albedo consistent with composite albedo; it should have a shading field that, outside the inserted fragment, is the same as the target scene's shading field; and composite surface normals are consistent with the final rendering's shading field. The result is a simple procedure that produces convincing and realistic shading. Moreover, our procedure does not require rendered images or image-decomposition from real images in the training or labeled annotations. In fact, our only use of simulated ground truth is our use of a pre-trained normal estimator. Qualitative results are strong, supported by a user study comparing against state-of-the-art image harmonization baseline.

Introduction

The "cut-and-paste" approach to image editing is conceptually simple but visually difficult. When an object is extracted from its original context and inserted into a new environment, it often appears as a "sticker"—a phenomenon caused by inconsistent shading. For instance, a plant photographed in direct sunlight will look jarringly out of place when pasted into a dimly lit interior.

Traditional solutions to this problem involve inverse rendering: reconstructing the object's 3D geometry and material properties, estimating the scene's lighting (environment map), and re-rendering the object. However, accurately recovering 3D shape and materials from a single image remains an unsolved challenge. Alternative "Image Harmonization" (IH) methods attempt to match colors but often degrade the object's original texture (albedo) while failing to resolve complex shading inconsistencies. This paper introduces DIPR (Deep Image Prior for Reshading), a method that leverages the inductive bias of neural networks to automatically adjust an object's shading to match its new environment without explicit 3D modeling.

Methodology: DIP as a Neural Renderer

Instead of explicit 3D modeling, DIPR treats a Deep Image Prior (DIP) network as a "neural renderer." The core idea is to optimize a U-Net architecture per image to satisfy physical consistency constraints.

Image Decomposition

The process begins by decomposing images into three intrinsic components: Albedo (the base color/texture), Shading (diffuse light), and Gloss (specular highlights). The decomposition network is trained in a self-supervised manner using statistical models of albedo and shading (Mondrian and Perlin noise), requiring no real-world ground truth.

Consistency Losses

The DIP is trained to generate a reshaded version of the inserted object by minimizing several key losses:

  1. Albedo Consistency: The reshaded object's albedo must remain faithful to the original source object.
  2. Background Consistency: Shading and gloss outside the inserted object's mask must match the target scene exactly.
  3. Normal Consistency: This is the critical component. Using a pre-trained, off-the-shelf normal estimator, the system ensures the new shading field is physically plausible relative to the object's estimated surface orientation.

To prevent the network from simply copying the source object, the authors employ Flipped Partial Convolution, forcing the network to "hallucinate" the surrounding scene and learn the global lighting context.

Datasets and Evaluation

To evaluate the system, the authors created a custom dataset featuring a real spherical lampshade photographed in various lighting conditions (hallways, stairwells, outdoors, and living rooms) to serve as ground truth.

The spherical lampshade dataset used for quantitative evaluation, showing ground truth across different lighting environments.

For qualitative testing, the method was applied to complex objects including Lego models, plants, and cars. Surprisingly, the authors found that state-of-the-art decomposition models optimized for standard benchmarks (like WHDR) often performed poorly for reshading because they smoothed out fine albedo details. DIPR prioritizes preserving high-frequency texture.

Key Results

Qualitative results demonstrate that DIPR handles complex materials—such as glossy car paint and thin plant leaves—far better than traditional image harmonization techniques.

Qualitative comparison of DIPR against Cut-and-Paste (CP) and Image Harmonization (IH) across various scenes and materials.

Quantitative and User Study

  • Metrics: On the spherical lampshade dataset, DIPR significantly outperformed IH baselines (DoveNet, RainNet) in PSNR and LPIPS.
  • User Study: In a study with over 100 participants, users were asked to distinguish real photos from DIPR renderings. The DIPR results were perceived as "real" 42.7% of the time, nearing the 50% threshold of a perfect "fooling" rate.

Discussion and Conclusion

DIPR offers a powerful, self-supervised alternative to traditional rendering for object insertion. By using surface normals as a latent variable and the DIP as a renderer, it achieves realistic lighting without perfect 3D meshes.

However, limitations remain. The per-image optimization is slow, taking roughly 15 minutes per render. Additionally, while the model adjusts shading on the object, it does not yet generate cast shadows from the object onto the environment. Future work may address these computational overheads and the integration of global shadow effects.

Summary updated

Abstract

We show how to insert an object from one image to another and get realistic results in the hard case, where the shading of the inserted object clashes with the shading of the scene. Rendering objects using an illumination model of the scene doesn't work, because doing so requires a geometric and material model of the object, which is hard to recover from a single image. In this paper, we introduce a method that corrects shading inconsistencies of the inserted object without requiring a geometric and physical model or an environment map. Our method uses a deep image prior (DIP), trained to produce reshaded renderings of inserted objects via consistent image decomposition inferential losses. The resulting image from DIP aims to have (a) an albedo similar to the cut-and-paste albedo, (b) a similar shading field to that of the target scene, and (c) a shading that is consistent with the cut-and-paste surface normals. The result is a simple procedure that produces convincing shading of the inserted object. We show the efficacy of our method both qualitatively and quantitatively for several objects with complex surface properties and also on a dataset of spherical lampshades for quantitative evaluation. Our method significantly outperforms an Image Harmonization (IH) baseline for all these objects. They also outperform the cut-and-paste and IH baselines in a user study with over 100 users.
BibTeX
@inproceedings{bhattad2020cut,
  author = {Anand Bhattad and David A. Forsyth},
  title = {{Cut-and-Paste Neural Rendering}},
  booktitle = {3DV},
  year = {2020},
}
Figure