Many-Worlds Inverse Rendering

Ziyi Zhang, Nicolas Roussel, Wenzel Jakob

Abstract: Discontinuous visibility changes remain a major bottleneck when optimizing surfaces within a physically-based inverse renderer. Many previous works have proposed sophisticated algorithms and data structures to sample visibility silhouettes more efficiently. Our work presents another solution: instead of differentiating a tentative surface locally, we differentiate a volumetric perturbation of a surface. We refer this as a many-worlds representation because it models a non-interacting superposition of conflicting explanations (worlds) of the input dataset. Each world is optically isolated from others, leading to a new transport law that distinguishes our method from prior work based on exponential random media. The resulting Monte Carlo algorithm is simpler and more efficient than prior methods. We demonstrate that our method promotes rapid convergence, both in terms of the total iteration count and the cost per iteration.

Introduction

The core challenge in physically-based inverse rendering is discontinuous visibility. When optimizing a 3D shape to match a 2D image, object silhouettes create mathematical discontinuities. A slight movement of a surface can cause a pixel to jump from background to foreground, making standard gradient-based optimization difficult.

Existing solutions typically fall into two categories: silhouette sampling, which is computationally expensive and brittle, or volumetric softening (similar to NeRF), which often results in "cloudy" geometry and slow rendering due to the complex Radiative Transfer Equation. "Many-Worlds Inverse Rendering" introduces a novel alternative by treating potential surface patches as non-interacting, mutually exclusive "worlds."

Methodology

The authors propose a Many-Worlds representation that considers an extended parameter space where every point has a probability of containing a surface patch.

Non-Interacting Superposition and Transport

Unlike traditional volume rendering where layers of fog shadow each other (transmittance), the Many-Worlds model treats every potential surface along a ray as a mutually exclusive explanation for the image. They do not shadow or scatter light into one another. This allows the derivation of a new derivative transport law that sums contributions from these "worlds," distributing gradients uniformly along a ray. This enables geometry to "materialize" in empty space without requiring a precise initial guess.

Comparison of local (a) and non-local (b) perturbations. The many-worlds approach (b) distributes derivatives along the entire light path, allowing for more robust optimization.

Scene Parameterization

The scene is parameterized by an occupancy field $\alpha(x)$ (probability of existence) and an orientation field $\beta(x)$ (the surface normal). The framework uses a hybrid approach: a discrete background surface $\bar{S}$ is used for the forward pass, while the many-worlds law is applied during the backward pass to update fields throughout the volume. This unifies "shading" and "boundary" derivatives into a single expression.

Results and Evaluation

The method was evaluated on complex geometries (e.g., Dragon, Heptoroid) and challenging scenarios involving specular and refractive paths, such as a mesh embedded in glass and viewed through a mirror.

Performance and Convergence

  • Speed: The method is significantly faster than volumetric approaches. In comparative tests, Many-Worlds achieved high-quality reconstruction in approximately 2 minutes, whereas anisotropic volume models took 16 minutes for lower-quality results.
  • Robustness: It successfully reconstructed complex shapes from empty space in 200 iterations, where SDF-based methods failed even after 2000 iterations.
  • Complex Light Paths: It is the first method to robustly handle "specular chains"—optimizing geometry seen only through reflections or refractions—without specialized silhouette-tracing code.
Geometry optimization of a scene containing complex light paths. The method successfully reconstructs text seen through a glass block and a mirror over 600 iterations.

Discussion and Limitations

While the Many-Worlds framework offers a simpler and more efficient Monte Carlo algorithm, it has limitations. The current implementation primarily extends derivatives to the exterior of the background surface, making it difficult to "hollow out" objects or create interior holes unless lighting is ideal. Additionally, the non-linear coupling between occupancy and orientation fields can sometimes be difficult to optimize. Future work may explore modeling a distribution of orientations at each point to represent micro-scale surface roughness.

Summary updated

Abstract

Discontinuous visibility changes remain a major bottleneck when optimizing surfaces within a physically-based inverse renderer. Many previous works have proposed sophisticated algorithms and data structures to sample visibility silhouettes more efficiently. Our work presents another solution: instead of differentiating a tentative surface locally, we differentiate a volumetric perturbation of a surface. We refer this as a many-worlds representation because it models a non-interacting superposition of conflicting explanations (worlds) of the input dataset. Each world is optically isolated from others, leading to a new transport law that distinguishes our method from prior work based on exponential random media. The resulting Monte Carlo algorithm is simpler and more efficient than prior methods. We demonstrate that our method promotes rapid convergence, both in terms of the total iteration count and the cost per iteration.
BibTeX
@misc{zhang2024many,
  author = {Ziyi Zhang and Nicolas Roussel and Wenzel Jakob},
  title = {{Many-Worlds Inverse Rendering}},
  year = {2024},
  eprint = {2408.16005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.16005},
}

Loading related content...

Figure