Abstract
We develop a method that recovers the surface, materials, and illumination of a scene from its posed multi-view images. In contrast to prior work, it does not require any additional data and can handle glossy objects or bright lighting. It is a progressive inverse rendering approach, which consists of three stages. First, we reconstruct the scene radiance and signed distance function (SDF) with our novel regularization strategy for specular reflections. Our approach considers both the diffuse and specular colors, which allows for handling complex view-dependent lighting effects for surface reconstruction. Second, we distill light visibility and indirect illumination from the learned SDF and radiance field using learnable mapping functions. Third, we design a method for estimating the ratio of incoming direct light represented via Spherical Gaussians reflected in a specular manner and then reconstruct the materials and direct illumination of the scene. Experimental results demonstrate that the proposed method outperforms the current state-of-the-art in recovering surfaces, materials, and lighting without relying on any additional data.
Introduction
Inverse rendering—the process of reconstructing a 3D scene's geometry, materials, and lighting from 2D images—is a fundamental task in computer vision. While existing methods perform well on Lambertian (matte) surfaces, glossy and highly reflective objects present significant challenges. Specular highlights, which shift across the surface with camera movement, often mislead implicit surface reconstruction algorithms like NeuS. These methods frequently introduce geometric artifacts, such as artificial bumps or concavities, to minimize color error at highlight locations. Furthermore, accurately disentangling an object's intrinsic albedo from reflected light remains difficult, often resulting in "baked-in" highlights in texture maps.
Methodology
Factored-NeuS addresses these challenges through a progressive three-stage inverse rendering pipeline:
- Stage 1: Geometry Reconstruction. The scene is represented using a Signed Distance Function (SDF). To prevent geometric deformations caused by specularities, the authors employ a joint rendering strategy. This approach supervises the network using both volume rendering and surface rendering, decomposing radiance into diffuse and specular components. This explicit modeling forces the network to explain highlights as view-dependent reflections rather than surface irregularities.
- Stage 2: Visibility and Indirect Illumination. With the geometry fixed, the model learns to predict light visibility (shadows) and indirect illumination. A differentiable integral representation for visibility is used, facilitating neural network training compared to non-differentiable binary ray-tracing.
- Stage 3: Material and Direct Lighting. The final stage recovers material properties using a Ward BRDF model and environment lighting represented by Spherical Gaussians. A key innovation is the learnable specular albedo parameter, which allows for precise estimation of the specular-to-diffuse reflection ratio, leading to cleaner material decomposition.
Results and Discussion
The proposed method was evaluated on several benchmarks, including SK3D, DTU, and the IndiSG dataset.
Geometry Reconstruction
Factored-NeuS demonstrates superior geometry recovery on glossy objects. On the SK3D dataset, it achieved a Chamfer Distance of 1.55, outperforming Geo-NeuS (1.81) despite the latter's use of additional point cloud data. Qualitatively, Factored-NeuS produces smooth, high-fidelity meshes, avoiding the "melted" or artifact-ridden surfaces produced by baselines.
Material Decomposition
In material decomposition tasks on the IndiSG dataset, the method achieved a Mean Albedo PSNR of 26.28, a significant improvement over the 23.63 achieved by the IndiSG baseline. This indicates a more accurate recovery of the object's true surface color without interference from lighting effects.
Conclusion
The "Anti-Bumpy" regularization and the use of a learnable specular albedo are central to the success of Factored-NeuS. By explicitly accounting for specular radiance, the model maintains a smooth SDF, preventing common concavity artifacts. While the method handles glossy surfaces exceptionally well, limitations remain in reconstructing extremely fine structures. Future work may focus on computational optimization and more complex BRDF models for anisotropic materials.
Summary updated