Abstract
Relit-NeuLF is a novel neural framework designed to achieve simultaneous high-quality relighting and novel view synthesis (NVS) in near real-time. By representing a scene as a 4D light field and decomposing rays into physical material properties (SVBRDF), the system can render complex scenes under new lighting conditions 140x faster than traditional volumetric methods like NeRF. This approach bridges the gap between computational efficiency and physical accuracy, making it suitable for demanding applications like 3D telepresence.
Introduction
Creating high-fidelity digital twins of real-world objects requires two primary capabilities: Novel View Synthesis (NVS), which allows viewing an object from new angles, and Relighting, which enables changing the lighting environment. Current state-of-the-art methods often face a significant "speed vs. quality" trade-off. Volumetric approaches like Neural Radiance Fields (NeRF) produce stunning results but are computationally expensive due to the necessity of "ray marching"—sampling many points along a ray. Furthermore, most NVS methods assume static lighting, while existing relighting methods often struggle with complex surface reflections or require dense, hard-to-acquire lighting data.
Relit-NeuLF addresses these challenges by utilizing a 4D light field representation that bypasses expensive volumetric sampling, enabling efficient rendering of complex materials like human skin under arbitrary illumination.
Methodology
The core innovation of Relit-NeuLF is a two-stage "analysis-synthesis" pipeline built on a 4D Light Field (NeuLF) backbone. Unlike NeRF’s 5D representation, this uses a two-plane parametrization ($u, v$ and $s, t$). Because the radiance along a ray is constant in empty space, mapping the ray's intersection with two planes is sufficient to determine its color, bypassing ray marching entirely.
The Analysis-Synthesis Pipeline
- DecomposeNet (Analysis): This MLP takes a 4D ray as input and predicts its Spatially-Varying Bidirectional Reflectance Distribution Function (SVBRDF) components: Albedo (base color), Normal (surface orientation), and Roughness (specularity).
- RenderNet (Synthesis): This MLP takes the predicted SVBRDF components plus a target light direction to synthesize the final pixel color. While the supervision is physics-informed, the final rendering is implicit, allowing the model to capture complex effects like subsurface scattering.
- Self-Supervised Learning: To overcome the difficulty of obtaining ground truth for surface normals or roughness, the authors employ a Microfacet Rendering Loss. This forces the network's output to be consistent with a physically-based rendering model, using physics as a teacher for the neural representation.
Experimental Setup and Datasets
The researchers evaluated Relit-NeuLF on both synthetic and real-world data. For synthetic evaluation, three complex objects (a face, a wooden train, and a mask) were rendered in Blender with 105 different light sources.
For real-world data, the authors constructed a custom capture rig featuring 13 synchronized DSLR cameras and 40 LED lights. They captured "One-Light-at-A-Time" (OLAT) data for four human subjects, creating a dataset that allows for the exploration of the 4D light field across both camera and light axes.
Key Results and Insights
Relit-NeuLF demonstrates a massive leap in efficiency, performing inference at approximately 150ms per frame. In comparison, a NeRF-based relighting baseline requires 22,143ms, representing a ~147x speedup.
Beyond speed, the method maintains high visual quality, outperforming state-of-the-art models like InvRender and Deep Relighting in PSNR and SSIM metrics. Qualitatively, the model produces sharper shadows and more realistic specular highlights. It also supports complex HDRI relighting, allowing subjects to be naturally integrated into new environments using HDR light probes.
Conclusion and Future Work
By moving from a 5D volume to a 4D light field and incorporating physics-informed priors, Relit-NeuLF achieves a robust representation that generalizes well to unseen light directions. While currently limited to non-frontal or "genus-zero" topologies (preventing 360-degree orbits), it is perfectly suited for portrait-style applications. Future work will explore more flexible light field representations to allow for unconstrained input illumination and finer surface detail capture.
Summary updated