Abstract
Inverse rendering aims to decompose a scene into its constituent components: shape, material, and lighting. However, this process is inherently ill-posed due to the ambiguity between these factors. We present Neural Incident Stokes Field (NeISF), a novel framework that leverages polarization cues to resolve these ambiguities. By modeling multi-bounce light as a polarized incident field represented by neural implicit functions, NeISF achieves high-fidelity reconstruction of geometry and materials, even in the presence of complex inter-reflections.
Introduction
Estimating the 3D properties of a scene from 2D images is a cornerstone of computer vision. While Neural Radiance Fields (NeRF) have revolutionized view synthesis, they often struggle with inverse rendering because they lack a physical model for light-matter interaction. Specifically, standard intensity-based cameras cannot easily distinguish between a dark material and a shadow, or between a physical bump and a specular reflection. This often results in "baked-in" lighting artifacts in the reconstructed geometry.
Polarization offers a powerful physical constraint to address these issues. The state of polarization changes predictably based on the surface normal and material type. However, existing polarimetric methods often assume single-bounce light, failing in realistic scenes where light bounces multiple times between surfaces.
Methodology: Neural Incident Stokes Fields
NeISF bridges the gap between wave optics and neural rendering by introducing an implicit representation for polarized light.
Implicit Stokes Representation
Instead of modeling scalar radiance, we represent light using the Stokes vector $\mathbf{s} = [I, Q, U, V]^T$, which captures the full polarization state. We use Multi-Layer Perceptrons (MLPs) to represent the incident Stokes field—the light arriving at any point in the scene from any direction.
Handling Multi-Bounce Inter-reflections
A key innovation of NeISF is treating all light hitting a surface as an "incident field." By training MLPs to record the Stokes vectors of the "second-to-last" bounce, we can simulate complex inter-reflections efficiently. This avoids the prohibitive computational cost of explicit path tracing during the optimization process.
The Implicit Rotation Trick
Polarization states are defined relative to a reference frame. Since diffuse and specular reflections require different frames, explicit geometric calculations can be expensive. NeISF employs an "implicit rotation" strategy where the MLPs are trained to output Stokes vectors that are already aligned with the required reference frames, simplifying the rendering pipeline.
Experimental Evaluation
We evaluated NeISF on both synthetic HDR datasets (rendered with Mitsuba 3.0) and real-world captures using a FLIR polarization camera.
Geometry and Material Accuracy
NeISF demonstrates superior performance in geometry reconstruction. On the synthetic Teapot dataset, NeISF achieved a Mean Angular Error (MAE) for surface normals of 1.502°, significantly outperforming previous state-of-the-art methods like PANDORA (5.809°) and VolSDF (5.943°).
Disentanglement and Material Editing
Qualitatively, NeISF excels at disentangling albedo from geometry. In real-world scenes, such as a teapot sitting on a book, NeISF correctly identifies text as a surface texture rather than a physical indentation. This clean separation enables downstream tasks like high-quality relighting and material editing, such as transforming a ceramic object into a metallic one while maintaining physical consistency.
Conclusion
NeISF introduces a robust framework for polarimetric inverse rendering by modeling the incident light field implicitly. By accounting for multi-bounce polarization, it provides a physically grounded way to reconstruct complex scenes. Future work includes extending the model to support metallic and translucent materials, as well as improving robustness to sensor noise in real-world captures.
Summary updated