Abstract
Cut-and-paste methods take an object from one image and insert it into another. Doing so often results in unrealistic looking images because the inserted object's shading is inconsistent with the target scene's shading. Existing reshading methods require a geometric and physical model of the inserted object, which is then rendered using environment parameters. Accurately constructing such a model only from a single image is beyond the current understanding of computer vision. We describe an alternative procedure -- cut-and-paste neural rendering, to render the inserted fragment's shading field consistent with the target scene. We use a Deep Image Prior (DIP) as a neural renderer trained to render an image with consistent image decomposition inferences. The resulting rendering from DIP should have an albedo consistent with composite albedo; it should have a shading field that, outside the inserted fragment, is the same as the target scene's shading field; and composite surface normals are consistent with the final rendering's shading field. The result is a simple procedure that produces convincing and realistic shading. Moreover, our procedure does not require rendered images or image-decomposition from real images in the training or labeled annotations. In fact, our only use of simulated ground truth is our use of a pre-trained normal estimator. Qualitative results are strong, supported by a user study comparing against state-of-the-art image harmonization baseline.
Introduction
The "cut-and-paste" approach to image editing is conceptually simple but visually difficult. When an object is extracted from its original context and inserted into a new environment, it often appears as a "sticker"—a phenomenon caused by inconsistent shading. For instance, a plant photographed in direct sunlight will look jarringly out of place when pasted into a dimly lit interior.
Traditional solutions to this problem involve inverse rendering: reconstructing the object's 3D geometry and material properties, estimating the scene's lighting (environment map), and re-rendering the object. However, accurately recovering 3D shape and materials from a single image remains an unsolved challenge. Alternative "Image Harmonization" (IH) methods attempt to match colors but often degrade the object's original texture (albedo) while failing to resolve complex shading inconsistencies. This paper introduces DIPR (Deep Image Prior for Reshading), a method that leverages the inductive bias of neural networks to automatically adjust an object's shading to match its new environment without explicit 3D modeling.
Methodology: DIP as a Neural Renderer
Instead of explicit 3D modeling, DIPR treats a Deep Image Prior (DIP) network as a "neural renderer." The core idea is to optimize a U-Net architecture per image to satisfy physical consistency constraints.
Image Decomposition
The process begins by decomposing images into three intrinsic components: Albedo (the base color/texture), Shading (diffuse light), and Gloss (specular highlights). The decomposition network is trained in a self-supervised manner using statistical models of albedo and shading (Mondrian and Perlin noise), requiring no real-world ground truth.
Consistency Losses
The DIP is trained to generate a reshaded version of the inserted object by minimizing several key losses:
- Albedo Consistency: The reshaded object's albedo must remain faithful to the original source object.
- Background Consistency: Shading and gloss outside the inserted object's mask must match the target scene exactly.
- Normal Consistency: This is the critical component. Using a pre-trained, off-the-shelf normal estimator, the system ensures the new shading field is physically plausible relative to the object's estimated surface orientation.
To prevent the network from simply copying the source object, the authors employ Flipped Partial Convolution, forcing the network to "hallucinate" the surrounding scene and learn the global lighting context.
Datasets and Evaluation
To evaluate the system, the authors created a custom dataset featuring a real spherical lampshade photographed in various lighting conditions (hallways, stairwells, outdoors, and living rooms) to serve as ground truth.
For qualitative testing, the method was applied to complex objects including Lego models, plants, and cars. Surprisingly, the authors found that state-of-the-art decomposition models optimized for standard benchmarks (like WHDR) often performed poorly for reshading because they smoothed out fine albedo details. DIPR prioritizes preserving high-frequency texture.
Key Results
Qualitative results demonstrate that DIPR handles complex materials—such as glossy car paint and thin plant leaves—far better than traditional image harmonization techniques.
Quantitative and User Study
- Metrics: On the spherical lampshade dataset, DIPR significantly outperformed IH baselines (DoveNet, RainNet) in PSNR and LPIPS.
- User Study: In a study with over 100 participants, users were asked to distinguish real photos from DIPR renderings. The DIPR results were perceived as "real" 42.7% of the time, nearing the 50% threshold of a perfect "fooling" rate.
Discussion and Conclusion
DIPR offers a powerful, self-supervised alternative to traditional rendering for object insertion. By using surface normals as a latent variable and the DIP as a renderer, it achieves realistic lighting without perfect 3D meshes.
However, limitations remain. The per-image optimization is slow, taking roughly 15 minutes per render. Additionally, while the model adjusts shading on the object, it does not yet generate cast shadows from the object onto the environment. Future work may address these computational overheads and the integration of global shadow effects.
Summary updated