Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering

Rui Li, Guangming Zang, Miao Qi, Wolfgang HeidricharXiv 2021 · OpenReview

TL;DR

This paper introduces a novel differentiable rendering framework that simultaneously reconstructs 3D geometry and surface reflectance from unconstrained multi-view images captured with standard cameras. This approach enables high-quality novel view synthesis and relighting in real-world scenarios without requiring controlled environments or specialized equipment.

What Problem Does This Paper Address?

Reconstructing the 3D shape and surface properties (like how light reflects off an object) of a scene from images is a fundamental challenge in computer vision. Doing this in "uncontrolled environments" – meaning everyday settings with varying lighting, camera angles, and without special setups – is particularly difficult. Previous methods often struggle with complex geometries, specular (shiny) reflections, and the unknown nature of real-world lighting. This paper aims to bridge this gap by enabling robust reconstruction and realistic rendering from casual, multi-view photographs.

Methods

The core of the proposed method is a differentiable rendering framework. This means they use a rendering engine (specifically, Mitsuba2) that can not only render an image from a 3D scene but also compute gradients of the rendering error with respect to the scene's parameters. This allows them to use gradient descent to optimize these parameters.

Here's a breakdown of their approach:

  • Scene Representation: They represent the scene using a mesh for geometry and per-vertex diffuse and specular reflectance properties. Environment lighting is modeled as a set of isotropic point sources.
  • Differentiable Rendering Pipeline: They leverage a physically-based rendering engine that supports differentiable operations. This engine simulates light transport, including diffuse and specular reflections, and allows backpropagation of errors.
  • Optimization: The system iteratively optimizes scene parameters (geometry, diffuse reflectance, specular reflectance, and lighting) by minimizing the photometric difference between rendered images and the input real-world images. To handle memory constraints and improve stability, they alternate between updating different parameter groups.
  • Image Formation Model: They use the standard rendering equation, incorporating diffuse and specular components modeled by the Cook-Torrance model with microfacet distributions.
  • Masking: To focus the reconstruction on the target object and avoid background noise, they use pre-computed binary masks derived from segmentation methods.

Datasets

The paper primarily uses custom captured scenes from hand-held cameras for their experiments. They also evaluate their method on the DTU dataset [26], a standard benchmark for multi-view stereo and 3D reconstruction.

Key Insights and Ideas

  • Differentiable Rendering for Uncontrolled Scenes: The key innovation is applying differentiable rendering to the challenging problem of reconstructing scenes from unconstrained, real-world photography. This moves beyond controlled lab settings.
  • Joint Optimization: Simultaneously optimizing geometry, reflectance, and lighting is crucial for achieving accurate and photo-realistic results, as these properties are highly interdependent.
  • Iterative Parameter Updates: The strategy of alternating updates for different scene parameters is a practical solution to manage memory limitations in GPU-intensive rendering and optimization processes.
  • Handling Complex Reflectance: The use of the Cook-Torrance model with microfacet distributions allows for the representation and reconstruction of both diffuse and specular surface properties, which are critical for realistic rendering.

Key Results

  • Visual Quality: The method produces visually compelling results, accurately reconstructing complex geometries, detailed textures, and realistic glossy reflections. Novel views generated by their approach are sharp and detailed.
  • Quantitative Performance: In comparisons with state-of-the-art methods like NeRF, Colmap, and PixelNeRF, their approach achieves superior performance in novel view synthesis, as measured by PSNR and SSIM metrics. For instance, on the "Fruit" dataset, their method achieved a PSNR of 29.02 and SSIM of 0.96, outperforming others.
  • Surface Refinement: The method can significantly correct local geometry errors that might arise from initial feature matching or reconstruction steps.
  • Relighting Capabilities: The reconstructed scene parameters enable photo-realistic relighting, allowing users to change lighting conditions and observe the effect on the scene.

Limitations and Future Work

  • Computational Cost: While they propose memory-efficient solutions, differentiable rendering and optimization remain computationally intensive, requiring significant processing time per iteration.
  • Initial Guess: The method relies on initial parameters (e.g., from COLMAP for geometry) and can be sensitive to the quality of this initial guess.
  • Segmentation Dependency: The reliance on pre-computed masks means the quality of reconstruction can be affected by the accuracy of the segmentation.

Future work could explore further optimizations for speed, reducing reliance on initial estimates, and extending the framework to handle more complex scene elements like transparency or volumetric effects.

Summary updated

Abstract

Simultaneous reconstruction of geometry and reflectance properties in uncontrolled environments remains a challenging problem. In this paper, we propose an efficient method to reconstruct the scene's 3D geometry and reflectance from multi-view photography using conventional hand-held cameras. Our method automatically builds a virtual scene in a differentiable rendering system that roughly matches the real world's scene parameters, optimized by minimizing photometric objectives alternatingly and stochastically. With the optimal scene parameters evaluated, photo-realistic novel views for various viewing angles and distances can then be generated by our approach. We present the results of captured scenes with complex geometry and various reflection types. Our method also shows superior performance compared to state-of-the-art alternatives in novel view synthesis visually and quantitatively.
BibTeX
@misc{li2021shape,
  author = {Rui Li and Guangming Zang and Miao Qi and Wolfgang Heidrich},
  title = {{Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering}},
  year = {2021},
  eprint = {2110.12975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.12975},
}
Figure