Abstract
We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene properties at that input location: volume density, surface normal, material parameters, distance to the first surface intersection in any direction, and visibility of the external environment in any direction. Together, these allow us to render novel views of the object under arbitrary lighting, including indirect illumination effects. The predicted visibility and surface intersection fields are critical to our model's ability to simulate direct and indirect illumination during training, because the brute-force techniques used by prior work are intractable for lighting conditions outside of controlled setups with a single light. Our method outperforms alternative approaches for recovering relightable 3D scene representations, and performs well in complex lighting settings that have posed a significant challenge to prior work.
Introduction
Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction by enabling photorealistic novel view synthesis. However, standard NeRF models "bake" the lighting into the scene's volume density and emitted color, preventing the user from changing the illumination. To achieve true relighting, a model must disentangle geometry, material properties, and the lighting environment.
The primary obstacle to this disentanglement is visibility. Determining if a point is lit requires checking for occlusions between that point and the light source. In a volumetric representation, this typically necessitates expensive raymarching for every light source at every pixel. For complex environments with thousands of light directions, this brute-force approach is computationally impossible to train.
Methods: Neural Visibility Fields
NeRV (Neural Reflectance and Visibility Fields) solves the visibility bottleneck by replacing expensive raymarching with a "shortcut" neural network.
Scene Representation
NeRV represents the scene using MLPs that output several key properties at any 3D location:
- Volume Density ($\sigma$): Defines the 3D structure.
- Surface Normal ($n$): Calculated analytically from the density field gradient for improved accuracy.
- Material Parameters: Includes diffuse albedo (base color) and specular roughness.
The Visibility Field
The core innovation is a separate MLP trained to act as a 3D lookup table. Given a 3D location and a direction, it directly predicts the visibility (light attenuation) and the expected termination depth (where a ray would hit a surface).
Crucially, this visibility field is supervised via a self-consistency loss rather than ground-truth shadow data. It is trained to be consistent with the density field: if the shape MLP places an object in a certain region, the visibility MLP learns that rays passing through that region are blocked.
Efficient Light Transport
By querying this visibility field, the system can efficiently calculate:
- Direct Illumination: Light coming directly from environment maps or point lights.
- One-Bounce Indirect Illumination: NeRV uses the predicted termination depth to find where a bounce originated, allowing it to simulate inter-reflections without tracing full secondary rays.
Experiments and Results
The authors evaluated NeRV on synthetic 3D scenes such as "Hotdogs," "Lego," and "Armadillo." Training was conducted under various conditions, including single point lights and complex ambient environment maps.
Performance and Realism
NeRV significantly outperformed prior state-of-the-art methods in complex lighting scenarios. While previous methods often failed when faced with multiple light sources during training, NeRV maintained high fidelity. On the "Ambient + Point" dataset, NeRV achieved a PSNR of ~25.1, compared to ~20.9 for the closest competitor.
Qualitatively, the model successfully captures subtle physical effects like inter-reflections—for example, the yellow glow inside the cab of a Lego bulldozer caused by light bouncing off its own surfaces. Because NeRV learns the underlying physics, it can render scenes under lighting conditions it has never seen before, such as a sunset environment map.
Conclusion
NeRV demonstrates that neural visibility fields can make complex light transport tractable within a volumetric framework. While currently limited to one-bounce indirect light and requiring known training lighting, it represents a significant step toward full inverse rendering from unconstrained image sets.
Summary updated