NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics

Chenhao Li, Taishi Ono, Takeshi Uemori, Sho Nitta, Hajime Mihara, Alexander Gatto, Hajime Nagahara, Yusuke Moriuchi

Introduction

Inverse rendering—the process of decomposing a 2D image into 3D geometry, material properties, and lighting—is a classic ill-posed problem in computer vision. While recent methods have leveraged polarization cues to reduce ambiguity, they often assume all objects are dielectrics (e.g., plastic or rubber). This assumption fails for conductors (metals), which interact with polarized light differently: they have complex refractive indices and lack diffuse polarization. Furthermore, the high specularity of metals often causes standard neural rendering methods to produce distorted geometry.

NeISF++ is the first polarized inverse rendering pipeline capable of accurately reconstructing scenes containing both conductors and dielectrics. It overcomes these challenges by introducing a general polarimetric material model and a novel geometry initialization technique based on the physics of polarization.

Methodology

The NeISF++ framework introduces two critical innovations to handle the diverse physical properties of real-world scenes.

General Polarimetric BRDF

The authors propose a general pBRDF that incorporates a binary indicator to toggle diffuse polarization on for dielectrics and off for conductors. Crucially, it implements a generalized Fresnel reflection term that supports complex refractive indices ($ior = \eta - ki$). This allows the model to mathematically represent the unique ways different metals reflect and polarize light, distinguishing between materials like gold and steel based on their polarization signatures.

Comparison of material decomposition between NeISF++ (Ours) and the baseline NeISF. Our method correctly estimates the complex refractive index (ior) and produces cleaner geometry and relighting for metallic objects.

DoLP-based Geometry Initialization

Standard methods often fail on metallic surfaces because intense specular highlights can "blind" the geometry optimization. NeISF++ instead uses the Degree of Linear Polarization (DoLP) for initial shape estimation. Since DoLP is invariant to light intensity but highly sensitive to surface orientation, it provides a stable physical cue that is not tricked by bright reflections, preventing the neural network from hallucinating geometry to explain away highlights.

Experiments and Results

The researchers evaluated NeISF++ using both synthetic and real-world HDR polarized datasets. The synthetic data includes complex objects like a "Helmet" and "Stanford Scan," while the real-world data features objects like metal spoons and pans captured with a FLIR polarization camera.

Quantitative Performance

NeISF++ significantly outperforms previous state-of-the-art methods such as NeISF, PANDORA, and NeRO. For instance, on the "Helmet" synthetic scene, NeISF++ achieved a mean angular error for surface normals of 1.789°, compared to 10.303° for the original NeISF. This improvement is particularly evident on highly specular surfaces where previous methods produced "foggy" or distorted results.

Quantitative and qualitative comparison of surface normal estimation on synthetic scenes. NeISF++ significantly reduces angular error compared to previous methods, especially on specular surfaces like the helmet visor.

Material Decomposition and Relighting

Beyond geometry, the model successfully separates diffuse albedo from roughness and correctly estimates the complex refractive index. This accurate decomposition allows for high-quality relighting, where reconstructed objects maintain their realistic metallic appearance when placed in new lighting environments, whereas earlier models often produced dull or distorted results.

Conclusion

NeISF++ advances the state of inverse rendering by explicitly modeling the distinct physics of conductors and dielectrics. By leveraging polarization-invariant cues for geometry and a generalized material model, it achieves unprecedented accuracy on challenging specular scenes. Future work will focus on automating the classification of material types to remove the current reliance on manual conductor-dielectric masks.

Summary updated

BibTeX
@inproceedings{li2025neisf,
  author = {Chenhao Li and Taishi Ono and Takeshi Uemori and Sho Nitta and Hajime Mihara and Alexander Gatto and Hajime Nagahara and Yusuke Moriuchi},
  title = {{NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02467},
}
Figure