Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects

Yue Fan, Ivan Skorokhodov, Oleg Voynov, Savva Ignatyev, Evgeny Burnaev, Peter Wonka, Yiqun Wang

Abstract

We develop a method that recovers the surface, materials, and illumination of a scene from its posed multi-view images. In contrast to prior work, it does not require any additional data and can handle glossy objects or bright lighting. It is a progressive inverse rendering approach, which consists of three stages. First, we reconstruct the scene radiance and signed distance function (SDF) with our novel regularization strategy for specular reflections. Our approach considers both the diffuse and specular colors, which allows for handling complex view-dependent lighting effects for surface reconstruction. Second, we distill light visibility and indirect illumination from the learned SDF and radiance field using learnable mapping functions. Third, we design a method for estimating the ratio of incoming direct light represented via Spherical Gaussians reflected in a specular manner and then reconstruct the materials and direct illumination of the scene. Experimental results demonstrate that the proposed method outperforms the current state-of-the-art in recovering surfaces, materials, and lighting without relying on any additional data.

Introduction

Inverse rendering—the process of reconstructing a 3D scene's geometry, materials, and lighting from 2D images—is a fundamental task in computer vision. While existing methods perform well on Lambertian (matte) surfaces, glossy and highly reflective objects present significant challenges. Specular highlights, which shift across the surface with camera movement, often mislead implicit surface reconstruction algorithms like NeuS. These methods frequently introduce geometric artifacts, such as artificial bumps or concavities, to minimize color error at highlight locations. Furthermore, accurately disentangling an object's intrinsic albedo from reflected light remains difficult, often resulting in "baked-in" highlights in texture maps.

Methodology

Factored-NeuS addresses these challenges through a progressive three-stage inverse rendering pipeline:

  1. Stage 1: Geometry Reconstruction. The scene is represented using a Signed Distance Function (SDF). To prevent geometric deformations caused by specularities, the authors employ a joint rendering strategy. This approach supervises the network using both volume rendering and surface rendering, decomposing radiance into diffuse and specular components. This explicit modeling forces the network to explain highlights as view-dependent reflections rather than surface irregularities.
  2. Stage 2: Visibility and Indirect Illumination. With the geometry fixed, the model learns to predict light visibility (shadows) and indirect illumination. A differentiable integral representation for visibility is used, facilitating neural network training compared to non-differentiable binary ray-tracing.
  3. Stage 3: Material and Direct Lighting. The final stage recovers material properties using a Ward BRDF model and environment lighting represented by Spherical Gaussians. A key innovation is the learnable specular albedo parameter, which allows for precise estimation of the specular-to-diffuse reflection ratio, leading to cleaner material decomposition.

Results and Discussion

The proposed method was evaluated on several benchmarks, including SK3D, DTU, and the IndiSG dataset.

Geometry Reconstruction

Factored-NeuS demonstrates superior geometry recovery on glossy objects. On the SK3D dataset, it achieved a Chamfer Distance of 1.55, outperforming Geo-NeuS (1.81) despite the latter's use of additional point cloud data. Qualitatively, Factored-NeuS produces smooth, high-fidelity meshes, avoiding the "melted" or artifact-ridden surfaces produced by baselines.

Qualitative comparison of geometry reconstruction. Factored-NeuS (Ours) recovers smooth surfaces without the artifacts seen in NeuS and Geo-NeuS.

Material Decomposition

In material decomposition tasks on the IndiSG dataset, the method achieved a Mean Albedo PSNR of 26.28, a significant improvement over the 23.63 achieved by the IndiSG baseline. This indicates a more accurate recovery of the object's true surface color without interference from lighting effects.

Recovered diffuse component on the IndiSG dataset, demonstrating successful material decomposition.

Conclusion

The "Anti-Bumpy" regularization and the use of a learnable specular albedo are central to the success of Factored-NeuS. By explicitly accounting for specular radiance, the model maintains a smooth SDF, preventing common concavity artifacts. While the method handles glossy surfaces exceptionally well, limitations remain in reconstructing extremely fine structures. Future work may focus on computational optimization and more complex BRDF models for anisotropic materials.

Summary updated

Abstract

We develop a method that recovers the surface, materials, and illumination of a scene from its posed multi-view images. In contrast to prior work, it does not require any additional data and can handle glossy objects or bright lighting. It is a progressive inverse rendering approach, which consists of three stages. First, we reconstruct the scene radiance and signed distance function (SDF) with our novel regularization strategy for specular reflections. Our approach considers both the diffuse and specular colors, which allows for handling complex view-dependent lighting effects for surface reconstruction. Second, we distill light visibility and indirect illumination from the learned SDF and radiance field using learnable mapping functions. Third, we design a method for estimating the ratio of incoming direct light represented via Spherical Gaussians reflected in a specular manner and then reconstruct the materials and direct illumination of the scene. Experimental results demonstrate that the proposed method outperforms the current state-of-the-art in recovering surfaces, materials, and lighting without relying on any additional data.
BibTeX
@inproceedings{fan2025factored,
  author = {Yue Fan and Ivan Skorokhodov and Oleg Voynov and Savva Ignatyev and Evgeny Burnaev and Peter Wonka and Yiqun Wang},
  title = {{Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01986},
}
Figure