Image-guided Neural Object Rendering

Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, Matthias NießnerICLR 2020 · OpenReview

Introduction

Traditional 3D reconstruction techniques excel at capturing an object's shape but often struggle to reproduce how it interacts with light. Specifically, capturing view-dependent effects—like the moving shine on a car's surface or the glint on a glass vase—remains a significant challenge.

Existing Image-Based Rendering (IBR) techniques attempt to solve this by warping captured photos onto a 3D proxy geometry. However, they typically suffer from two main issues:

  1. Ghosting Artifacts: Imperfect 3D geometry causes blended images to misalign, creating blurry, double-edge artifacts.
  2. Specular Inconsistency: Standard blending treats moving highlights as static textures. As the camera moves, these highlights either "pop" unnaturally or blend into dull, unrealistic surfaces.

To address this, the authors propose a hybrid rendering technique that bridges the gap between classical computer graphics and GAN-based image synthesis. By explicitly disentangling view-dependent effects from the underlying diffuse appearance, their system synthesizes photo-realistic novel views that remain temporally stable and visually sharp.

Methodology

Unlike many neural renderers that attempt to memorize an object's appearance from scratch, this approach is image-guided. It uses original captured photos as a high-frequency detail source, leveraging deep learning primarily as an intelligent blender and lighting estimator. The authors propose a two-stage neural pipeline:

Proposed rendering pipeline

1. EffectsNet: Disentangling Light and Color

The first stage utilizes a Siamese neural network called EffectsNet to "un-shade" the input images. Using a 3D proxy geometry (reconstructed via multi-view stereo tools like COLMAP), EffectsNet predicts the view-dependent lighting effects (specularities) of the captured views.

By subtracting these predicted effects from the original images, the network isolates a "diffuse" version of the object. Because diffuse surfaces look the same from any angle, these cleaned images are significantly easier to warp and blend without introducing artifacts.

2. CompositionNet: Re-rendering the Novel View

Once the diffuse images are warped into the new target view, the pipeline recalculates and re-inserts the appropriate view-dependent effects for that specific camera angle. Finally, a U-Net-style composition network blends these multiple reprojected images together. This network acts as a sophisticated corrector—fixing reprojection errors, inpainting missing regions, and ensuring the final output is a cohesive, photo-realistic image.

Key Insights

  • Disentanglement is Key: The core innovation is recognizing that shiny images shouldn't be warped directly. Separating the "shine" from the "color" bypasses the mathematical mess of interpolating moving highlights.
  • Self-Supervised Consistency: EffectsNet is trained using a consistency loss. If the network correctly subtracts the view-dependent effects from two different views, the resulting diffuse images should look identical when projected to the same location. This clever formulation allows the model to learn without requiring impossible-to-capture "ground truth" diffuse images.
  • Network Capacity Efficiency: Because the network relies on actual source images for texture, it doesn't waste parameter capacity memorizing pixels. Instead, it focuses entirely on understanding light transport and geometry-image alignment.

Datasets and Results

The researchers evaluated their method on both synthetic data (generated via the Mitsuba Renderer to provide perfect ground truth) and real-world data (videos captured with a Nikon D5300 and reconstructed via COLMAP).

The proposed method significantly outperformed both classical IBR and pure deep learning baselines (like Pix2Pix):

  • Quantitative: On synthetic data, the Mean Squared Error (MSE) was roughly 3x to 10x lower than traditional IBR methods. For instance, on a synthetic "Car" sequence, the method achieved an MSE of 3.61, compared to 12.62 for Pix2Pix and 39.90 for classical IBR.
  • Qualitative: The outputs feature much sharper textures and temporally stable highlights. While baseline methods produce highlights that "crawl" or flicker distractingly across frames, this image-guided approach produces smooth, realistic reflections that naturally follow camera movement.

Limitations and Future Work

While highly effective, the current pipeline has a few limitations. First, it requires object-specific training; the model must be trained from scratch for every new object (taking roughly an hour), which limits its use in instant AR applications. Second, it maintains a dependency on geometry. While the composition network can fix small geometric errors, a catastrophic failure in the initial 3D reconstruction (e.g., on a highly transparent object) will cause the rendering to fail.

Future work will likely explore "category-specific" models—such as a network pre-trained to understand the general geometry and reflectance of cars—to reduce or eliminate the need for lengthy per-object training.

Summary updated

Abstract

We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours and sightseeing, the digital inspection of historical artifacts). A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object. As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.
BibTeX
@inproceedings{thies2020image,
  author = {Justus Thies and Michael Zollhöfer and Christian Theobalt and Marc Stamminger and Matthias Nießner},
  title = {{Image-guided Neural Object Rendering}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=rPomRbYh16q},
}
Figure