Abstract
Radiance fields are powerful and, hence, popular models for representing the appearance of complex scenes. Yet, constructing them based on image observations gives rise to ambiguities and uncertainties. We propose a versatile approach for learning Gaussian radiance fields with explicit and fine-grained uncertainty estimates that impose only little additional cost compared to uncertainty-agnostic training. Our key observation is that uncertainties can be modeled as a low-dimensional manifold in the space of radiance field parameters that is highly amenable to Monte Carlo sampling. Importantly, our uncertainties are differentiable and, thus, allow for gradient-based optimization of subsequent captures that optimally reduce ambiguities. We demonstrate state-of-the-art performance on next-best-view planning tasks, including high-dimensional illumination planning for optimal radiance field relighting quality.
Introduction
Reconstructing a 3D scene from 2D images is a classic "ill-posed" inverse problem. When input data is sparse or viewing angles are limited, many different 3D configurations can explain the same set of images. This leads to epistemic uncertainty—the model’s lack of knowledge about regions it hasn't seen or cannot resolve.
While radiance fields like 3D Gaussian Splatting (3DGS) have achieved remarkable visual quality, quantifying their uncertainty remains a significant challenge. Existing methods typically face a trilemma: they are either too computationally expensive to scale to millions of parameters, produce coarse and noisy estimates, or lack the differentiability required for downstream optimization tasks. This paper introduces a novel manifold sampling strategy that addresses all three challenges, enabling efficient, high-quality, and differentiable uncertainty estimation.
The Manifold Sampling Approach
The core innovation of this work is the realization that the uncertainty of a radiance field—despite involving millions of parameters—can be effectively captured on a remarkably low-dimensional manifold.
Stochastic Radiance Fields
The authors build their method on 3D Gaussian Splatting. Instead of treating the radiance field parameters $\theta$ as fixed values, they model them as a distribution. To keep this computationally tractable, they define the parameters as a linear function of a low-dimensional latent variable $z$:
$$G(z) = \bar{\theta} + \hat{B}z$$
In this model, $\bar{\theta}$ represents the mean parameter values, while $\hat{B}$ is a low-rank matrix that defines a $k$-dimensional parallelotope (a multi-dimensional parallelogram) in the high-dimensional parameter space.
Efficiency through Dimensionality Reduction
One of the most striking findings of the research is that a rank as low as $k=2$ is often sufficient to capture the relevant uncertainty. By modeling the uncertainty volume as this low-rank manifold, the authors can use Monte Carlo sampling with very few samples (often just $M=1$ during training) to achieve stable gradients. This results in a training process that is only about 14% slower than standard, uncertainty-agnostic 3DGS, making it highly practical for real-world applications.
Differentiable Planning
Because the uncertainty estimate is a simple, direct function of the rendered mean, it is "trivially differentiable." This allows for gradient-based optimization in active planning tasks. Rather than selecting the "next best view" from a discrete set of candidates, researchers can use standard optimizers like Adam to "walk" a virtual camera or light source through continuous space to find the mathematically optimal position to reduce scene ambiguity.
Experimental Results
The authors evaluated their approach across several standard benchmarks, including NeRF Synthetic, Mip-NeRF360, and the LF dataset.
Next-Best-View Planning
In tasks designed to determine where to place a camera next to improve model quality, this method consistently outperformed state-of-the-art baselines such as ActiveNeRF and FisherRF. The ability to perform continuous optimization allowed the model to find superior viewpoints that discrete candidate-selection methods would likely miss.
Active Illumination Planning
The paper also introduces the novel task of active illumination planning. In relightable scenes, the model can determine which lighting conditions will most effectively resolve ambiguities in material properties (like shininess or roughness). This is the first time differentiable uncertainty has been applied to optimize illumination in radiance fields.
Discussion and Limitations
While the manifold sampling approach provides a significant leap in efficiency and utility, there are areas for further exploration:
- Architectural Bias: The current implementation is optimized for 3D Gaussian Splatting. While the theory is generalizable to NeRFs, the inherent smoothness of coordinate-based networks might alter the behavior of the uncertainty field.
- Discrete Artifacts: On certain metrics like the Area Under Sparsification Error (AUSE), some NeRF-based methods maintain a slight edge. This is likely due to the discrete nature of 3DGS, which can produce "popping" artifacts that the uncertainty model may interpret as noise.
- Priors: The authors found that a uniform prior outperformed the traditional Gaussian prior. They argue that for occluded regions, a uniform prior better represents total ignorance, whereas a Gaussian prior incorrectly assumes a central probability.
Conclusion
By modeling radiance field uncertainty as a low-dimensional linear manifold, this work enables a new class of differentiable active vision tasks. The minimal computational overhead and the ability to use standard gradient-based optimization make it a versatile tool for robotics, digital twinning, and high-fidelity 3D reconstruction.
Summary updated