Abstract
The problem of inferring object shape from a single 2D image is underconstrained. Prior knowledge about what objects are plausible can help, but even given such prior knowledge there may still be uncertainty about the shapes of occluded parts of objects. Recently, conditional neural radiance field (NeRF) models have been developed that can learn to infer good point estimates of 3D models from single 2D images. The problem of inferring uncertainty estimates for these models has received less attention. In this work, we propose probabilistic NeRF (ProbNeRF), a model and inference strategy for learning probabilistic generative models of 3D objects' shapes and appearances, and for doing posterior inference to recover those properties from 2D images. ProbNeRF is trained as a variational autoencoder, but at test time we use Hamiltonian Monte Carlo (HMC) for inference. Given one or a few 2D images of an object (which may be partially occluded), ProbNeRF is able not only to accurately model the parts it sees, but also to propose realistic and diverse hypotheses about the parts it does not see. We show that key to the success of ProbNeRF are (i) a deterministic rendering scheme, (ii) an annealed-HMC strategy, (iii) a hypernetwork-based decoder architecture, and (iv) doing inference over a full set of NeRF weights, rather than just a low-dimensional code.
Introduction
Inferring a 3D object from a single 2D image is a classic "ill-posed" problem in computer vision. When observing only one side of an object, such as the front of a car, the configuration of occluded parts remains uncertain. While recent Conditional Neural Radiance Field (NeRF) models produce impressive point estimates, they typically fail to represent this inherent uncertainty. In critical applications like robotics or autonomous driving, quantifying uncertainty is as vital as the prediction itself. ProbNeRF addresses this by proposing a diverse range of plausible 3D hypotheses rather than a single "best guess."
Methods
ProbNeRF treats 3D reconstruction as a Bayesian inference problem, utilizing a Variational Autoencoder (VAE) framework for training and Hamiltonian Monte Carlo (HMC) for test-time inference.
The architecture incorporates several innovations:
- Hypernetwork Decoder: Instead of the standard NeRF approach of concatenating a latent code to every 3D coordinate, ProbNeRF uses a hypernetwork that takes the latent code and outputs the entire set of weights for a small NeRF. This significantly reduces rendering costs, enabling the iterative sampling required for HMC.
- Deterministic "Foam" Renderer: Standard NeRFs use stochastic sampling, which creates noisy gradients. ProbNeRF employs a deterministic renderer that treats space as a lattice of surfaces, providing the smooth gradients necessary for efficient MCMC sampling.
- Annealed HMC Inference: To capture the full diversity of the posterior, the model uses HMC with a temperature annealing strategy. This allows the sampler to explore multiple modes—different plausible shapes—without becoming trapped in local optima.
- Weight Perturbation: Inference is performed over both low-dimensional latent codes and small perturbations to the raw NeRF weights, allowing the model to adapt to novel objects outside its learned prior.
Datasets and Results
The model was evaluated on two benchmarks: SRN Cars and GHUM (a dataset of 60,000 high-fidelity human body scans). The latter was specifically used to test the model's ability to handle complex pose uncertainty under occlusion.
ProbNeRF demonstrates superior diversity compared to standard Variational Inference (VI) and point-estimate baselines. By switching to HMC at test time, the model can "hallucinate" various valid versions of unseen parts, such as different limb positions for a partially occluded human or geometric variations for the back of a car. Despite its probabilistic nature, the individual samples maintain high fidelity, often appearing sharper and more realistic than those from deterministic baselines.
Limitations and Future Work
While the hypernetwork and deterministic renderer improve efficiency, HMC remains an iterative and computationally intensive process. This latency currently limits ProbNeRF's use in real-time applications. Additionally, the complexity of the pipeline—integrating normalizing flows, hypernetworks, and HMC—presents implementation challenges. Future research will focus on scaling these probabilistic methods to complex, multi-object scenes with dynamic backgrounds.
Summary updated