Relit-NeuLF: Efficient Relighting and Novel View Synthesis via Neural 4D Light Field

Zhong Li, Liangchen Song, Zhang Chen, Xiangyu Du, Lele Chen, Junsong Yuan, Yi Xu

Abstract

Relit-NeuLF is a novel neural framework designed to achieve simultaneous high-quality relighting and novel view synthesis (NVS) in near real-time. By representing a scene as a 4D light field and decomposing rays into physical material properties (SVBRDF), the system can render complex scenes under new lighting conditions 140x faster than traditional volumetric methods like NeRF. This approach bridges the gap between computational efficiency and physical accuracy, making it suitable for demanding applications like 3D telepresence.

Introduction

Creating high-fidelity digital twins of real-world objects requires two primary capabilities: Novel View Synthesis (NVS), which allows viewing an object from new angles, and Relighting, which enables changing the lighting environment. Current state-of-the-art methods often face a significant "speed vs. quality" trade-off. Volumetric approaches like Neural Radiance Fields (NeRF) produce stunning results but are computationally expensive due to the necessity of "ray marching"—sampling many points along a ray. Furthermore, most NVS methods assume static lighting, while existing relighting methods often struggle with complex surface reflections or require dense, hard-to-acquire lighting data.

Relit-NeuLF addresses these challenges by utilizing a 4D light field representation that bypasses expensive volumetric sampling, enabling efficient rendering of complex materials like human skin under arbitrary illumination.

Methodology

The core innovation of Relit-NeuLF is a two-stage "analysis-synthesis" pipeline built on a 4D Light Field (NeuLF) backbone. Unlike NeRF’s 5D representation, this uses a two-plane parametrization ($u, v$ and $s, t$). Because the radiance along a ray is constant in empty space, mapping the ray's intersection with two planes is sufficient to determine its color, bypassing ray marching entirely.

The Relit-NeuLF architecture: A 4D ray is decomposed into SVBRDF components (Albedo, Normal, Roughness) before being synthesized into a final color based on target light direction.

The Analysis-Synthesis Pipeline

  1. DecomposeNet (Analysis): This MLP takes a 4D ray as input and predicts its Spatially-Varying Bidirectional Reflectance Distribution Function (SVBRDF) components: Albedo (base color), Normal (surface orientation), and Roughness (specularity).
  2. RenderNet (Synthesis): This MLP takes the predicted SVBRDF components plus a target light direction to synthesize the final pixel color. While the supervision is physics-informed, the final rendering is implicit, allowing the model to capture complex effects like subsurface scattering.
  3. Self-Supervised Learning: To overcome the difficulty of obtaining ground truth for surface normals or roughness, the authors employ a Microfacet Rendering Loss. This forces the network's output to be consistent with a physically-based rendering model, using physics as a teacher for the neural representation.

Experimental Setup and Datasets

The researchers evaluated Relit-NeuLF on both synthetic and real-world data. For synthetic evaluation, three complex objects (a face, a wooden train, and a mask) were rendered in Blender with 105 different light sources.

The capture setup and 4D data organization, showing how subjects are captured across varying camera viewpoints and light directions.

For real-world data, the authors constructed a custom capture rig featuring 13 synchronized DSLR cameras and 40 LED lights. They captured "One-Light-at-A-Time" (OLAT) data for four human subjects, creating a dataset that allows for the exploration of the 4D light field across both camera and light axes.

Key Results and Insights

Relit-NeuLF demonstrates a massive leap in efficiency, performing inference at approximately 150ms per frame. In comparison, a NeRF-based relighting baseline requires 22,143ms, representing a ~147x speedup.

Beyond speed, the method maintains high visual quality, outperforming state-of-the-art models like InvRender and Deep Relighting in PSNR and SSIM metrics. Qualitatively, the model produces sharper shadows and more realistic specular highlights. It also supports complex HDRI relighting, allowing subjects to be naturally integrated into new environments using HDR light probes.

Conclusion and Future Work

By moving from a 5D volume to a 4D light field and incorporating physics-informed priors, Relit-NeuLF achieves a robust representation that generalizes well to unseen light directions. While currently limited to non-frontal or "genus-zero" topologies (preventing 360-degree orbits), it is perfectly suited for portrait-style applications. Future work will explore more flexible light field representations to allow for unconstrained input illumination and finer surface detail capture.

Summary updated

BibTeX
@inproceedings{li2023relit,
  author = {Zhong Li and Liangchen Song and Zhang Chen and Xiangyu Du and Lele Chen and Junsong Yuan and Yi Xu},
  title = {{Relit-NeuLF: Efficient Relighting and Novel View Synthesis via Neural 4D Light Field}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612160},
}
Figure