Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects

Yue Fan, Ningjing Fan, Ivan Skorokhodov, Oleg Voynov, Savva Ignatyev, Evgeny Burnaev, Peter Wonka, Yiqun Wang

TL;DR

Factored-NeuS is a three-stage inverse rendering framework that reconstructs high-quality geometry, materials, and lighting from multi-view images. By jointly optimizing volume and surface rendering and introducing a learnable specular albedo, it effectively eliminates the "bumpy" geometric artifacts that typically plague the reconstruction of glossy objects.

What Problem Does This Paper Address?

Reconstructing 3D scenes from images is a cornerstone of computer vision, but glossy surfaces remain a major hurdle. Standard implicit surface methods (like NeuS) often fail because specular highlights change depending on where you stand. To explain these view-dependent color shifts, neural networks often "cheat" by creating fake geometric dents or bumps—essentially trying to hide the highlight inside a concave deformation.

Furthermore, disentangling a pixel's color into its base material (diffuse albedo), its shininess (specular component), and the environmental lighting is an under-determined problem. Existing methods often "leak" specular highlights into the diffuse component, resulting in inaccurate material maps that look like they have highlights painted onto them.

Figure 1. Left: Geometry visualization for NeuS, Geo-NeuS and our method on the Pot scene from SK3D dataset. Existing surface reconstruction methods struggle to recover the correct geometry of glossy objects due to the complex view-dependent effects they induce. The weak color model of these methods compels to represent such effects through concave geometric deformations rather than proper view-dependent radiance, leading to shape artifacts. In contrast, our method can correctly reconstruct a highly reflective surface due to our joint appearance, diffuse, and specular color training strategy. Right: Visualization of the recovered diffuse color component on the Bunny scene from DTU for IndiSG [56] and our method. Existing inverse rendering methods overestimate the diffuse material component in the presence of specular highlights. Our regularization strategy allows us to properly disentangle the color into diffuse and specular components.

Methods

Factored-NeuS employs a progressive three-stage pipeline to factorize the scene:

  1. Stage 1: Geometry Reconstruction. The authors propose a joint rendering strategy. They use volume rendering (standard in NeRF-style models) alongside surface rendering. Crucially, they decompose the radiance into diffuse and specular components during this stage. By forcing the volume-rendered color and the surface-rendered color to match, they regularize the Signed Distance Function (SDF), preventing the formation of geometric artifacts caused by highlights.
  2. Stage 2: Visibility and Indirect Light. Using the frozen SDF from Stage 1, the model learns to predict light visibility and indirect illumination. Instead of binary "shadow or no shadow" ray-tracing, they use a continuous visibility function derived from the SDF weights, which is much easier for neural networks to optimize.
  3. Stage 3: Material and Direct Light. The final stage recovers the BRDF (using the Ward model) and direct environment lighting (represented by Spherical Gaussians). A key novelty here is the specular albedo network ($M_{sa}$), which learns a per-point ratio of specular reflection, allowing for much more accurate material decomposition on complex real-world surfaces.
Figure 2. Overview for Stage 1 (left), Stage 2 (mid), and Stage 3 (right) training pipelines. The first stage (left) trains the SDF network Sθ which outputs a feature vector vf ∈R256, SDF value s ∈R, and normal n ∈S2 (as a normalized gradient of s; denoted via the dashed line); diffuse and specular surface color networks Md and Ms produce their respective colors cd, cs ∈R3 via surface rendering, which are then combined through tone mapping γ(·) to get the final surface color Csur ∈R3; volumetric color network Mc produces the volumetrically rendered color Cvol ∈R3. The ref operation denotes computation of the reflection direction ωr ∈S2 from normal n and ray direction ω ∈S2. In the second stage (mid), indirect light network Mind and light visibility network Mν produce their respective indirect light SGs and light visibility ν. In the third stage (right), we optimize the material BRDF auto-encoder with the sparsity constraint [56], our novel specular albedo network Msa, and the indirect illumination network Mind. See Sec 3 for details.

Datasets

The authors evaluated their method on a mix of real and synthetic datasets:

  • SK3D: A real-world dataset featuring highly reflective objects with complex lighting.
  • DTU: A standard multi-view benchmark for 3D reconstruction.
  • Shiny: A synthetic dataset specifically designed to test glossy surface reconstruction.
  • IndiSG: A dataset with ground-truth material maps (albedo, roughness) used to evaluate decomposition accuracy.
  • Glossy: A dataset containing both real and synthetic scenes.

Key Insights and Ideas

  • Joint Rendering as Regularization: The most significant "aha moment" is that surface rendering handles specular reflections better, while volume rendering is more robust for general shape. Using them jointly forces the SDF to remain smooth even in the presence of intense highlights.
  • Continuous Visibility: Moving from discrete ray-traced visibility to a continuous integral representation based on SDF weights makes the visibility function differentiable and much more stable for network training.
  • Learnable Specular Albedo: By not assuming a uniform specular response across the object, the $M_{sa}$ network allows the model to distinguish between parts of an object that are truly glossy and those that are more matte, leading to cleaner albedo maps.

Key Results

Factored-NeuS achieves state-of-the-art results across multiple benchmarks:

  • Geometry: On the glossy scenes of the SK3D dataset, it achieved a mean Chamfer distance of 1.55, significantly outperforming NeuS (2.31) and NeRO (3.65).
  • Material Decomposition: On the IndiSG dataset, it reached a mean PSNR for diffuse albedo of 21.21, compared to IndiSG's 22.47 and NVDiffrec's 13.56.
  • Qualitative: As shown in the figures, the method produces clean, smooth surfaces for objects like metal pots and ceramic jugs where previous methods produced "melted" or dented geometry.
Figure 3. Qualitative comparison on DTU and SK3D datasets.

Limitations and Future Work

Despite its strengths, the method has some limitations:

  • Fine Structures: It still struggles to reconstruct very small, intricate details like the tiny nails on the back of a chair or fine textures on a pillow.
  • Fresnel Effects: While increasing the Fresnel value helps with chrome-like materials, it can sometimes lead to a loss of fine texture details in the albedo.
  • Future Directions: The authors suggest that future work should focus on better decomposition strategies for fine-grained structures and more adaptive handling of Fresnel reflections.

Summary updated

Abstract

We develop a method that recovers the surface, materials, and illumination of a scene from its posed multi-view images. In contrast to prior work, it does not require any additional data and can handle glossy objects or bright lighting. It is a progressive inverse rendering approach, which consists of three stages. In the first stage, we reconstruct the scene radiance and signed distance function (SDF) with a novel regularization strategy for specular reflections. We propose to explain a pixel color using both surface and volume rendering jointly, which allows for handling complex view-dependent lighting effects for surface reconstruction. In the second stage, we distill light visibility and indirect illumination from the learned SDF and radiance field using learnable mapping functions. Finally, we design a method for estimating the ratio of incoming direct light reflected in a specular manner and use it to reconstruct the materials and direct illumination. Experimental results demonstrate that the proposed method outperforms the current state-of-the-art in recovering surfaces, materials, and lighting without relying on any additional data.
BibTeX
@inproceedings{fan2025factored,
  author = {Yue Fan and Ningjing Fan and Ivan Skorokhodov and Oleg Voynov and Savva Ignatyev and Evgeny Burnaev and Peter Wonka and Yiqun Wang},
  title = {{Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01986},
}
Figure