Introduction
Inverse rendering—the process of decomposing a 2D image into 3D geometry, material properties, and lighting—is a classic ill-posed problem in computer vision. While recent methods have leveraged polarization cues to reduce ambiguity, they often assume all objects are dielectrics (e.g., plastic or rubber). This assumption fails for conductors (metals), which interact with polarized light differently: they have complex refractive indices and lack diffuse polarization. Furthermore, the high specularity of metals often causes standard neural rendering methods to produce distorted geometry.
NeISF++ is the first polarized inverse rendering pipeline capable of accurately reconstructing scenes containing both conductors and dielectrics. It overcomes these challenges by introducing a general polarimetric material model and a novel geometry initialization technique based on the physics of polarization.
Methodology
The NeISF++ framework introduces two critical innovations to handle the diverse physical properties of real-world scenes.
General Polarimetric BRDF
The authors propose a general pBRDF that incorporates a binary indicator to toggle diffuse polarization on for dielectrics and off for conductors. Crucially, it implements a generalized Fresnel reflection term that supports complex refractive indices ($ior = \eta - ki$). This allows the model to mathematically represent the unique ways different metals reflect and polarize light, distinguishing between materials like gold and steel based on their polarization signatures.
DoLP-based Geometry Initialization
Standard methods often fail on metallic surfaces because intense specular highlights can "blind" the geometry optimization. NeISF++ instead uses the Degree of Linear Polarization (DoLP) for initial shape estimation. Since DoLP is invariant to light intensity but highly sensitive to surface orientation, it provides a stable physical cue that is not tricked by bright reflections, preventing the neural network from hallucinating geometry to explain away highlights.
Experiments and Results
The researchers evaluated NeISF++ using both synthetic and real-world HDR polarized datasets. The synthetic data includes complex objects like a "Helmet" and "Stanford Scan," while the real-world data features objects like metal spoons and pans captured with a FLIR polarization camera.
Quantitative Performance
NeISF++ significantly outperforms previous state-of-the-art methods such as NeISF, PANDORA, and NeRO. For instance, on the "Helmet" synthetic scene, NeISF++ achieved a mean angular error for surface normals of 1.789°, compared to 10.303° for the original NeISF. This improvement is particularly evident on highly specular surfaces where previous methods produced "foggy" or distorted results.
Material Decomposition and Relighting
Beyond geometry, the model successfully separates diffuse albedo from roughness and correctly estimates the complex refractive index. This accurate decomposition allows for high-quality relighting, where reconstructed objects maintain their realistic metallic appearance when placed in new lighting environments, whereas earlier models often produced dull or distorted results.
Conclusion
NeISF++ advances the state of inverse rendering by explicitly modeling the distinct physics of conductors and dielectrics. By leveraging polarization-invariant cues for geometry and a generalized material model, it achieves unprecedented accuracy on challenging specular scenes. Future work will focus on automating the classification of material types to remove the current reliance on manual conductor-dielectric masks.
Summary updated