Introduction
Neural Radiance Fields (NeRFs) have revolutionized novel view synthesis by representing scenes as continuous volumetric functions. However, a significant limitation of standard NeRFs is that lighting information is typically "baked" into the scene's appearance. This prevents the relighting of captured objects under novel illumination conditions, a critical requirement for applications in virtual production and augmented reality.
Existing "relightable NeRF" methods often rely on the distant light assumption, where light sources are modeled as being at infinity. While mathematically convenient, this assumption fails in nearfield scenarios—such as a studio setup—where the proximity of light sources causes significant variations in light direction and intensity across the subject. Furthermore, many high-fidelity relighting techniques require complex hardware like Light Stages. ReNeRF addresses these challenges by modeling nearfield light transport using a simplified studio setup.
Methodology
ReNeRF draws inspiration from Image-Based Relighting (IBRL), specifically the One-Light-At-A-Time (OLAT) technique. In this approach, an object is captured under multiple individual light sources. By linearly combining these captures, one can simulate any lighting environment.
Nearfield Modeling and the OLAT MLP
Unlike previous models that assume parallel light rays, ReNeRF explicitly accounts for the distance between 3D points in the volume and the light sources. It models area lights as collections of point lights and incorporates the $1/d^2$ inverse-square law for light attenuation.
The architecture consists of two primary components:
- NeRF MLP: Predicts the geometry (density) and spatial features of the scene.
- OLAT MLP: A specialized network that predicts how a specific 3D point responds to light from a given direction.
Spherical Codebook
To enable relighting from arbitrary directions beyond the discrete set of training lights, ReNeRF employs a spherical codebook. This codebook is parameterized by Spherical Harmonics (SH), allowing the model to interpolate smoothly between captured light positions. This effectively "super-resolves" a limited number of physical lights into a continuous lighting basis. Additionally, the model disentangles diffuse and specular components, ensuring that highlights shift realistically with changes in camera or light position.
Experimental Setup
The researchers evaluated ReNeRF using both synthetic and real-world datasets. The real-world setup involved 10 synchronized video cameras and 32 rectangular LED bars. The subjects—including a furry toy dog, wax candles, and human heads—were chosen to test the model's ability to handle complex light transport effects such as subsurface scattering, self-occlusion, and translucency.
Results and Discussion
ReNeRF demonstrates superior performance compared to state-of-the-art baselines like NeRFactor and NeRF-SHL. Quantitatively, the model achieved PSNR scores between 31.17 and 38.14 on validation sets.
Qualitatively, the model excels at capturing material-specific lighting effects. For instance, it accurately renders the "glow" of light passing through thin structures like human ears or wax, a feat that traditional albedo-based models struggle to achieve. Furthermore, the model generalizes well to novel lighting directions, producing realistic shadows and highlights even for light positions far outside the training arc.
Conclusion
ReNeRF provides a robust framework for nearfield relighting in NeRFs using accessible studio hardware. By combining the principles of IBRL with volumetric neural representations, it achieves high-fidelity results that respect the physics of nearfield light transport. While current rendering speeds remain a challenge for real-time applications, future integration with accelerated NeRF backbones offers a promising path forward.
Summary updated