Introduction
Inverse rendering—the process of decomposing 2D images into 3D shape, material properties, and lighting—is a fundamental challenge in computer vision. While recent neural rendering techniques have made significant strides, they often struggle with scenes containing metallic surfaces. Most existing methods assume objects are dielectrics (like plastic or glass), ignoring the unique physical properties of conductors (metals). NeISF++ addresses this gap by leveraging polarization cues to accurately reconstruct scenes containing both material types.
The Challenge of Metallic Surfaces
Standard inverse rendering is notoriously ill-posed, as many combinations of shape and light can produce the same pixel values. Conductors exacerbate this problem in two ways:
- Complex Refractive Index: Unlike dielectrics, metals require a complex refractive index ($ior = \eta - ki$) to model their appearance accurately.
- Specular Highlights: Metals are often highly reflective. These intense highlights can "blind" standard algorithms, leading to distorted geometry and unrealistic material reconstructions.
Methodology
NeISF++ introduces a unified polarimetric material model and a novel geometry initialization technique to overcome these hurdles.
Unified pBRDF
The framework utilizes a general Polarized Bidirectional Reflectance Distribution Function (pBRDF) that handles both conductors and dielectrics. It incorporates a binary indicator to toggle diffuse polarization—which exists in dielectrics but is physically absent in conductors—and implements a general Fresnel reflection term supporting complex refractive indices.
DoLP-based Geometry Initialization
Instead of initializing 3D shapes (Signed Distance Fields) using raw light intensity, which is prone to error near bright highlights, NeISF++ uses the Degree of Linear Polarization (DoLP). DoLP is invariant to light intensity and depends primarily on surface orientation, providing a stable physical cue for the initial geometry.
Joint Optimization
The system jointly optimizes the geometry, material fields (roughness, albedo, and complex IOR), and lighting (represented as an Incident Stokes Field) through a self-supervised loss against captured polarized images.
Experimental Results
NeISF++ demonstrates a dramatic improvement over previous state-of-the-art methods like the original NeISF and PANDORA, particularly in material fidelity and relighting accuracy.
On synthetic benchmarks, NeISF++ achieved a mean angular error for surface normals of 1.789°, a significant reduction from the 10.303° error seen in the original NeISF. By correctly estimating the imaginary part of the refractive index, the method ensures that reconstructed metals look metallic and glossy rather than dull and plastic-like.
Conclusion and Future Work
NeISF++ represents a significant step forward in neural inverse rendering by bridging the gap between dielectric and conductor modeling. While the current implementation requires a manual conductor-dielectric mask, future research aims to automate this process using data-driven segmentation or error-driven optimization to further streamline the reconstruction of complex, real-world scenes.
Summary updated