NeISF: Neural Incident Stokes Field for Geometry and Material Estimation

Chenhao Li, Taishi Ono, Takeshi Uemori, Hajime Mihara, Alexander Gatto, Hajime Nagahara, Yusuke Moriuchi

Abstract

Inverse rendering aims to decompose a scene into its constituent components: shape, material, and lighting. However, this process is inherently ill-posed due to the ambiguity between these factors. We present Neural Incident Stokes Field (NeISF), a novel framework that leverages polarization cues to resolve these ambiguities. By modeling multi-bounce light as a polarized incident field represented by neural implicit functions, NeISF achieves high-fidelity reconstruction of geometry and materials, even in the presence of complex inter-reflections.

Introduction

Estimating the 3D properties of a scene from 2D images is a cornerstone of computer vision. While Neural Radiance Fields (NeRF) have revolutionized view synthesis, they often struggle with inverse rendering because they lack a physical model for light-matter interaction. Specifically, standard intensity-based cameras cannot easily distinguish between a dark material and a shadow, or between a physical bump and a specular reflection. This often results in "baked-in" lighting artifacts in the reconstructed geometry.

Polarization offers a powerful physical constraint to address these issues. The state of polarization changes predictably based on the surface normal and material type. However, existing polarimetric methods often assume single-bounce light, failing in realistic scenes where light bounces multiple times between surfaces.

Methodology: Neural Incident Stokes Fields

NeISF bridges the gap between wave optics and neural rendering by introducing an implicit representation for polarized light.

Implicit Stokes Representation

Instead of modeling scalar radiance, we represent light using the Stokes vector $\mathbf{s} = [I, Q, U, V]^T$, which captures the full polarization state. We use Multi-Layer Perceptrons (MLPs) to represent the incident Stokes field—the light arriving at any point in the scene from any direction.

Handling Multi-Bounce Inter-reflections

A key innovation of NeISF is treating all light hitting a surface as an "incident field." By training MLPs to record the Stokes vectors of the "second-to-last" bounce, we can simulate complex inter-reflections efficiently. This avoids the prohibitive computational cost of explicit path tracing during the optimization process.

The NeISF pipeline integrates Signed Distance Fields (SDF), BRDF fields, and Incident Stokes fields into a differentiable polarimetric renderer.

The Implicit Rotation Trick

Polarization states are defined relative to a reference frame. Since diffuse and specular reflections require different frames, explicit geometric calculations can be expensive. NeISF employs an "implicit rotation" strategy where the MLPs are trained to output Stokes vectors that are already aligned with the required reference frames, simplifying the rendering pipeline.

Experimental Evaluation

We evaluated NeISF on both synthetic HDR datasets (rendered with Mitsuba 3.0) and real-world captures using a FLIR polarization camera.

Geometry and Material Accuracy

NeISF demonstrates superior performance in geometry reconstruction. On the synthetic Teapot dataset, NeISF achieved a Mean Angular Error (MAE) for surface normals of 1.502°, significantly outperforming previous state-of-the-art methods like PANDORA (5.809°) and VolSDF (5.943°).

Quantitative comparison of surface normal estimation on the Bunny and Teapot datasets. NeISF significantly outperforms state-of-the-art methods in accuracy and detail.

Disentanglement and Material Editing

Qualitatively, NeISF excels at disentangling albedo from geometry. In real-world scenes, such as a teapot sitting on a book, NeISF correctly identifies text as a surface texture rather than a physical indentation. This clean separation enables downstream tasks like high-quality relighting and material editing, such as transforming a ceramic object into a metallic one while maintaining physical consistency.

Conclusion

NeISF introduces a robust framework for polarimetric inverse rendering by modeling the incident light field implicitly. By accounting for multi-bounce polarization, it provides a physically grounded way to reconstruct complex scenes. Future work includes extending the model to support metallic and translucent materials, as well as improving robustness to sensor noise in real-world captures.

Summary updated

Abstract

Multi-view inverse rendering is the problem of estimating the scene parameters such as shapes, materials, or illuminations from a sequence of images captured under different viewpoints. Many approaches, however, assume single light bounce and thus fail to recover challenging scenarios like inter-reflections. On the other hand, simply extending those methods to consider multi-bounced light requires more assumptions to alleviate the ambiguity. To address this problem, we propose Neural Incident Stokes Fields (NeISF), a multi-view inverse rendering framework that reduces ambiguities using polarization cues. The primary motivation for using polarization cues is that it is the accumulation of multi-bounced light, providing rich information about geometry and material. Based on this knowledge, the proposed incident Stokes field efficiently models the accumulated polarization effect with the aid of an original physically-based differentiable polarimetric renderer. Lastly, experimental results show that our method outperforms the existing works in synthetic and real scenarios.
BibTeX
@inproceedings{li2023neisf,
  author = {Chenhao Li and Taishi Ono and Takeshi Uemori and Hajime Mihara and Alexander Gatto and Hajime Nagahara and Yusuke Moriuchi},
  title = {{NeISF: Neural Incident Stokes Field for Geometry and Material Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure