Manifold Sampling for Differentiable Uncertainty in Radiance Fields

Linjie Lyu, Ayush Tewari, Marc Habermann, Shunsuke Saito, Michael Zollhöfer, Thomas Leimkühler, Christian Theobalt

TL;DR

This paper introduces a novel "manifold sampling" strategy to estimate differentiable uncertainty in radiance fields (specifically 3D Gaussian Splatting). By modeling parameter uncertainties as a low-dimensional linear manifold, the authors enable efficient training and—crucially—gradient-based optimization for active planning tasks like choosing the next best camera viewpoint or illumination condition.

What Problem Does This Paper Address?

Reconstructing a 3D scene from 2D images is an "ill-posed" inverse problem: many different 3D configurations can explain the same set of images, especially when the input data is sparse. This leads to epistemic uncertainty—the model doesn't know what it doesn't know.

While existing methods (like NeRF-W or FisherRF) attempt to quantify this, they often face a trilemma:

  1. Quality: Estimates are often too coarse or noisy.
  2. Efficiency: Modeling the full covariance of millions of radiance field parameters is computationally impossible.
  3. Differentiability: Most current methods don't allow you to "backpropagate" through the uncertainty. If you want to find the best spot for a new camera, you usually have to pick from a pre-defined list of candidates rather than using optimization to find the mathematically optimal pose.

Methods

The authors propose Stochastic Radiance Fields built on 3D Gaussian Splatting (3DGS). Their key innovation is Manifold Sampling:

  • Low-Rank Manifold: Instead of a full covariance matrix (which would be millions of entries squared), they model the uncertainty volume as a $k$-dimensional parallelotope (a multi-dimensional parallelogram) in the high-dimensional parameter space.
  • The Model: They define the parameters $\theta$ as a linear function of a low-dimensional latent variable $z$: $G(z) = \bar{\theta} + \hat{B}z$. Here, $\bar{\theta}$ is the mean and $\hat{B}$ is a low-rank matrix.
  • Efficient Sampling: Because the manifold is low-dimensional (they find $k=2$ is often sufficient!), they can use Monte Carlo sampling with very few samples (even just $M=1$ during training) to get stable, high-quality gradients.
  • Differentiable Planning: Because the uncertainty estimate is a simple function of the rendered mean, it is "trivially differentiable." This allows researchers to use standard optimizers (like Adam) to move a "virtual camera" or "virtual light" around a scene to find the position that maximizes expected information gain.

Datasets

The authors evaluated their approach on several standard benchmarks:

  • NeRF Synthetic: A collection of 8 synthetic objects with complex geometry and materials.
  • Mip-NeRF360: Real-world, unbounded scenes used to test the limits of reconstruction in complex environments.
  • LF Dataset: Used specifically for the Area Under Sparsification Error (AUSE) metric to correlate uncertainty with depth error.
  • Relightable Scenes: Synthetic scenes used to demonstrate the novel task of active illumination planning.

Key Insights and Ideas

  • The "Aha!" Moment: The most surprising finding is that the uncertainty of a radiance field—despite having millions of parameters—can be effectively captured on a manifold with a rank as low as 2. This massive dimensionality reduction is what makes the method practical.
  • Uniform > Gaussian: The authors argue for a uniform prior over the traditional Gaussian prior. They note that for things like depth or occluded regions, a Gaussian prior incorrectly assumes a "center" of probability, whereas a uniform prior better represents total ignorance.
  • Derivative-Free Estimates: Unlike Laplace-based methods that require complex higher-order derivatives, this sampling approach is "derivative-free" regarding the model parameters, which simplifies the implementation and improves stability.

Key Results

  • Minimal Overhead: The method is only about 14% slower to train than a standard, uncertainty-agnostic 3DGS model, making it highly practical for real-world use.
  • Superior Planning: In "Next-Best-View" tasks, the method consistently outperformed state-of-the-art baselines like ActiveNeRF and FisherRF.
  • Continuous Optimization: The authors demonstrated that they could start a camera from a random position and use gradients to "walk" it to an optimal viewpoint. This is a significant step up from "candidate selection" methods that are limited to a fixed grid of possible views.
  • Illumination Planning: The paper is the first to demonstrate differentiable illumination planning, showing that the model can determine which lighting conditions will most effectively resolve ambiguities in a scene's material properties.

Limitations and Future Work

  • 3DGS Bias: The current implementation is tied to 3D Gaussian Splatting. While the theory applies to NeRFs, the authors note that NeRF's inherent smoothness might change how the uncertainty field behaves.
  • Depth Correlation: On certain metrics (like AUSE), some NeRF-based methods still hold a slight edge, likely because 3DGS is a discrete representation that can produce "popping" artifacts that confuse pixel-wise uncertainty.
  • Future Directions: The authors suggest extending manifold sampling to other signal modalities (like lidar or thermal) and exploring more complex, non-linear manifolds for even higher-fidelity uncertainty.

Summary updated

Abstract

Radiance fields are powerful and, hence, popular models for representing the appearance of complex scenes. Yet, constructing them based on image observations gives rise to ambiguities and uncertainties. We propose a versatile approach for learning Gaussian radiance fields with explicit and fine-grained uncertainty estimates that impose only little additional cost compared to uncertainty-agnostic training. Our key observation is that uncertainties can be modeled as a low-dimensional manifold in the space of radiance field parameters that is highly amenable to Monte Carlo sampling. Importantly, our uncertainties are differentiable and, thus, allow for gradient-based optimization of subsequent captures that optimally reduce ambiguities. We demonstrate state-of-the-art performance on next-best-view planning tasks, including high-dimensional illumination planning for optimal radiance field relighting quality.
BibTeX
@inproceedings{lyu2024manifold,
  author = {Linjie Lyu and Ayush Tewari and Marc Habermann and Shunsuke Saito and Michael Zollhöfer and Thomas Leimkühler and Christian Theobalt},
  title = {{Manifold Sampling for Differentiable Uncertainty in Radiance Fields}},
  booktitle = {SIGGRAPH Asia},
  year = {2024},
  doi = {10.1145/3680528.3687655},
}

Loading related content...

Figure