Abstract
We show how to insert an object from one image to another and get realistic results in the hard case, where the shading of the inserted object clashes with the shading of the scene. Rendering objects using an illumination model of the scene doesn't work, because doing so requires a geometric and material model of the object, which is hard to recover from a single image. In this paper, we introduce a method that corrects shading inconsistencies of the inserted object without requiring a geometric and physical model or an environment map. Our method uses a deep image prior (DIP), trained to produce reshaded renderings of inserted objects via consistent image decomposition inferential losses. The resulting image from DIP aims to have (a) an albedo similar to the cut-and-paste albedo, (b) a similar shading field to that of the target scene, and (c) a shading that is consistent with the cut-and-paste surface normals. The result is a simple procedure that produces convincing shading of the inserted object. We show the efficacy of our method both qualitatively and quantitatively for several objects with complex surface properties and also on a dataset of spherical lampshades for quantitative evaluation. Our method significantly outperforms an Image Harmonization (IH) baseline for all these objects. They also outperform the cut-and-paste and IH baselines in a user study with over 100 users.
Introduction
The "cut-and-paste" paradigm is the simplest way to create new images, but achieving realism is notoriously difficult. When an object is moved from one environment to another, its internal shading—the interaction of light and shadow across its surface—often clashes with the new scene's lighting, making the insertion look "fake." Existing solutions typically fall into two categories: Image Harmonization (IH), which adjusts global color and brightness but fails to change the direction or quality of light, and Inverse Rendering, which attempts to reconstruct full 3D geometry and lighting—a highly ill-posed problem from a single 2D image. This paper proposes a middle ground: an image-based approach that "reshades" objects to be structurally consistent with the target scene using only 2D pixel information.
Methodology
The core of the proposed approach is DIPR (Deep Image Prior for Reshading). It utilizes a neural network (specifically a U-Net) as a renderer, optimizing the network's weights for each specific insertion task rather than training on a large dataset. The system decomposes images into three layers: Albedo (base color), Shading (diffuse light), and Gloss (specular highlights).
To prevent the network from simply copying the original shading, the authors employ "Flipped Partial Convolution," forcing the model to learn the relationship between the object's surface and the scene's light by "outpainting" the scene from the object's context. Optimization is driven by consistency losses ensuring the albedo matches the original object, the shading/gloss outside the object matches the target scene, and the resulting shading is physically plausible according to pre-trained surface normal estimators.
Datasets and Evaluation
Evaluation was performed using several datasets: MS-COCO for base decomposition training, ADE20K for shading consistency, and a custom Lampshade Dataset. The latter consists of 25 real photos of a spherical lampshade in various rooms to serve as ground truth for quantitative assessment. Additionally, complex objects like plants, chairs, and vehicles were used to test performance on diverse materials including matte, glossy, and specular surfaces.
Results and Discussion
DIPR demonstrates significant improvements over baselines. On the Lampshade Dataset, it achieved a PSNR of 39.53, outperforming the state-of-the-art Image Harmonization baseline (DoveNet) at 35.57. In a user study with over 100 participants, DIPR renderings were mistaken for real photographs 42.7% of the time, approaching the 50% threshold of total indistinguishability.
A key insight is the trade-off between shading and albedo; keeping high-frequency details in the albedo layer actually aids the reshading process. Furthermore, the model exhibits an emergent understanding of 3D shape, correctly rendering a flat white disk as a sphere to maintain consistency with surrounding light and estimated normals.
Conclusion
While DIPR produces high-quality results, it currently faces limitations such as high computational cost (approx. 15 minutes per insertion) and the lack of contact shadow generation. Future work may address these issues and reduce the dependency on the accuracy of external surface normal estimators. Despite these challenges, DIPR represents a significant step toward realistic, automated image composition.
Summary updated