NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics

Chenhao Li, Taishi Ono, Takeshi Uemori, Sho Nitta, Hajime Mihara, Alexander Gatto, Hajime Nagahara, Yusuke Moriuchi

Introduction

Inverse rendering—the process of decomposing 2D images into 3D shape, material properties, and lighting—is a fundamental challenge in computer vision. While recent neural rendering techniques have made significant strides, they often struggle with scenes containing metallic surfaces. Most existing methods assume objects are dielectrics (like plastic or glass), ignoring the unique physical properties of conductors (metals). NeISF++ addresses this gap by leveraging polarization cues to accurately reconstruct scenes containing both material types.

The Challenge of Metallic Surfaces

Standard inverse rendering is notoriously ill-posed, as many combinations of shape and light can produce the same pixel values. Conductors exacerbate this problem in two ways:

  1. Complex Refractive Index: Unlike dielectrics, metals require a complex refractive index ($ior = \eta - ki$) to model their appearance accurately.
  2. Specular Highlights: Metals are often highly reflective. These intense highlights can "blind" standard algorithms, leading to distorted geometry and unrealistic material reconstructions.

Methodology

NeISF++ introduces a unified polarimetric material model and a novel geometry initialization technique to overcome these hurdles.

The NeISF++ pipeline jointly optimizes geometry (SDF), material properties (BRDF fields), and incident lighting (Stokes fields) using a polarimetric physics-based rendering loss.

Unified pBRDF

The framework utilizes a general Polarized Bidirectional Reflectance Distribution Function (pBRDF) that handles both conductors and dielectrics. It incorporates a binary indicator to toggle diffuse polarization—which exists in dielectrics but is physically absent in conductors—and implements a general Fresnel reflection term supporting complex refractive indices.

DoLP-based Geometry Initialization

Instead of initializing 3D shapes (Signed Distance Fields) using raw light intensity, which is prone to error near bright highlights, NeISF++ uses the Degree of Linear Polarization (DoLP). DoLP is invariant to light intensity and depends primarily on surface orientation, providing a stable physical cue for the initial geometry.

Joint Optimization

The system jointly optimizes the geometry, material fields (roughness, albedo, and complex IOR), and lighting (represented as an Incident Stokes Field) through a self-supervised loss against captured polarized images.

Experimental Results

NeISF++ demonstrates a dramatic improvement over previous state-of-the-art methods like the original NeISF and PANDORA, particularly in material fidelity and relighting accuracy.

Comparison between NeISF++ (Ours) and the original NeISF. NeISF++ successfully recovers the complex refractive index (real and imaginary parts) and produces superior relighting results for metallic objects.

On synthetic benchmarks, NeISF++ achieved a mean angular error for surface normals of 1.789°, a significant reduction from the 10.303° error seen in the original NeISF. By correctly estimating the imaginary part of the refractive index, the method ensures that reconstructed metals look metallic and glossy rather than dull and plastic-like.

Conclusion and Future Work

NeISF++ represents a significant step forward in neural inverse rendering by bridging the gap between dielectric and conductor modeling. While the current implementation requires a manual conductor-dielectric mask, future research aims to automate this process using data-driven segmentation or error-driven optimization to further streamline the reconstruction of complex, real-world scenes.

Summary updated

BibTeX
@inproceedings{li2025neisf,
  author = {Chenhao Li and Taishi Ono and Takeshi Uemori and Sho Nitta and Hajime Mihara and Alexander Gatto and Hajime Nagahara and Yusuke Moriuchi},
  title = {{NeISF++: Neural Incident Stokes Field for Polarized Inverse Rendering of Conductors and Dielectrics}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02467},
}
Figure