Standard Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) are master mimics but poor editors. They achieve photorealism by mapping a 3D coordinate and a viewing direction directly to a color and density value. While this works beautifully for novel view synthesis, it creates a fundamental flaw: lighting-geometry entanglement.
Because standard neural scene representations only memorize the final radiant light leaving a point, the environmental illumination—including shadows, specular highlights, and reflections—is permanently "baked" into the scene's geometry and appearance. If you attempt to change the light source or move an object to a new environment, the original shadows remain painted onto the surfaces like dark stains, and highlights fail to shift realistically.
To solve this, a major research direction focuses on inverse rendering: decomposing the single baked radiance value back into its constituent physical components. By factoring the scene into independent representations for geometry, material properties, and illumination, these methods transition neural rendering from a passive capture tool into an editable, relightable graphics pipeline.
Factoring the Rendering Equation
To achieve true relighting, neural networks must learn to solve the rendering equation implicitly or explicitly. Instead of training a single network to output color from position and view direction, the problem is factored into three distinct components:
- Geometry: The underlying 3D structure, represented via volumetric density, Signed Distance Fields (SDFs), or 3D Gaussian primitives.
- Materials: The intrinsic surface properties, typically parameterized via a Bidirectional Reflectance Distribution Function (BRDF) that defines albedo (base color), roughness, and specularity.
- Illumination: The incident light field, often modeled via environment maps, spherical harmonics, or latent light probes.
Standard NeRF: (Position, View Direction) ──> [ Neural Network ] ──> Baked Color
Inverse NeRF: (Position) ──> [ Geometry, BRDF, Illumination Fields ] ──> Differentiable Rendering ──> Relightable Color
A primary challenge in this factorization is ambiguity: a dark region in an image could be a dark material (low albedo) or a bright material sitting in a shadow (low illumination).
To resolve this, Weicai Ye et al. introduced IntrinsicNeRF, which integrates intrinsic image decomposition directly into the volumetric rendering pipeline. By enforcing constraints that split appearance into reflectance, shading, and a view-dependent residual, the model learns to isolate base material colors from external lighting. This separation allows for intuitive editing tasks like surface recoloring.
Similarly, Bingchen Gong et al. leverage gradient propagation to separate albedo from illumination, enabling users to stitch together different neural scene components captured under completely different environmental lighting without creating jarring visual seams.
Modeling Light Transport and Occlusion
Knowing the surface material is only half the battle; a relightable model must also understand how light travels through space and where it gets blocked to cast shadows.
Standard neural networks struggle to generalize to unobserved lighting conditions because they rely on absolute spatial coordinates, failing to understand the relative geometric relationship between a light source and a surface point. To bridge this gap, Marco Toschi et al. proposed an architecture that explicitly decouples visibility (shadows) from the surface BRDF by modeling the relative positions of light sources and query points. This allows the model to apply learned light-interaction principles to entirely novel light configurations.
Calculating full light transport and self-occlusion in real-time is computationally expensive. To make inverse rendering practical, Jesus Zarzar et al. adapted the "split sum approximation"—a technique widely used in real-time game engines—into neural fields. By utilizing specialized MLPs to handle pre-integrated lighting and self-occlusion separately, their method extracts high-quality geometry and materials in a fraction of the traditional training time.
For complex indoor scenes where light bounces multiple times (global illumination), researchers have integrated physics-based ray tracing into neural architectures. Jingsen Zhu et al. combined neural Signed Distance Fields (SDFs) with differentiable Monte Carlo ray tracing to accurately simulate indirect lighting and material swapping.
To achieve similar high-fidelity light transport at interactive framerates, HONGZE CHEN et al. introduced GI-GS, a framework that applies deferred shading and path tracing to 3D Gaussian Splatting, allowing point-based radiance fields to dynamically update shadows and reflections on the fly.
Handling Complex Media and Physical Cues
The challenge of baked-in illumination intensifies when dealing with non-opaque or complex materials. Translucent objects scatter light beneath their surfaces, meaning their appearance depends on a complex web of internal light paths.
Quan Zheng et al. addressed this for participating media (such as fog, clouds, or wax) by combining Monte Carlo ray tracing for direct light with a learned Spherical Harmonics field to approximate multiple scattering events efficiently. To bypass the immense computational burden of simulating individual light bounces in complex materials, Hong-Xing Yu et al. proposed Object-Centric Neural Scattering Functions (OSFs). OSFs treat subsurface light transport as a "cumulative radiance transfer" function, enabling free-viewpoint relighting of translucent objects from ordinary 2D images.
Sometimes, standard intensity images do not contain enough information to resolve the ambiguity between geometry and complex reflections. To overcome this, Chenhao Li et al. introduced NeISF, which leverages polarization cues. By implicitly modeling how polarized light bounces through a scene as a Neural Incident Stokes Field, the authors successfully disentangle inter-reflections that typically cause standard NeRF pipelines to fail.
Bringing Relighting to Dynamic Assets
The ultimate validation of decoupled illumination is the ability to animate an asset and place it into a completely novel virtual environment. This is particularly vital for human avatars and garments.
Wenzhang Sun et al. developed a self-supervised framework that reconstructs relightable 3D human models from monocular video by mapping the appearance into a canonical space defined by spatially varying BRDFs. This ensures that as the avatar takes on new poses, its clothing and skin react dynamically to external light sources.
To achieve interactive rendering speeds for these avatars, He Wang et al. paired the efficiency of 3D Gaussian Splatting with a mesh-based visibility technique, delivering real-time rendering with full shadow generation. Extending this concept to physics engines, Michelle Guo et al. developed Physics-Based Gaussian Cloth (PGC). By embedding 3D Gaussians into a physics-simulatable mesh and separating fabric material properties from the environment, the reconstructed garments can be realistically simulated, animated, and relit inside any standard virtual environment.
Key Foundational Work
NeRF for Outdoor Scene Relighting (2022) — contributed 1 concept elements to this direction.
Neural Relightable Participating Media Rendering (2021) — contributed 1 concept elements to this direction.
HDR-NeRF: High Dynamic Range Neural Radiance Fields (2022) — contributed 1 concept elements to this direction.