Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects

Yue Fan, Ningjing Fan, Ivan Skorokhodov, Oleg Voynov, Savva Ignatyev, Evgeny Burnaev, Peter Wonka, Yiqun Wang

TL;DR

This paper introduces Factored-NeuS, a novel three-stage inverse rendering framework that reconstructs object geometry, materials, and illumination from multi-view images, specifically excelling at handling glossy objects and challenging lighting conditions without requiring extra data. It achieves this by jointly optimizing surface and volume rendering, and by introducing a learnable specular albedo to better disentangle diffuse and specular reflections.

What Problem Does This Paper Address?

Reconstructing 3D scenes, including their shape, material properties, and lighting, from a set of 2D images is a fundamental challenge in computer vision with applications ranging from virtual reality to robotics. While neural radiance fields (NeRFs) and implicit neural surfaces have made significant progress, existing methods often struggle with glossy objects. Glossy surfaces produce highlights that can be misinterpreted as geometric artifacts by surface reconstruction methods. Furthermore, specular reflections can "leak" into diffuse components, leading to inaccurate material and lighting decomposition. Many prior works also rely heavily on synthetic data, which may not translate well to real-world scenarios. Factored-NeuS aims to address these limitations by developing a robust method for reconstructing glossy objects from real-world multi-view images.

Methods

Factored-NeuS employs a progressive, three-stage approach:

  • Stage 1: Surface Reconstruction from Glossy Appearance: This stage focuses on reconstructing the scene's signed distance function (SDF) and radiance. A key innovation is the joint optimization of surface and volume rendering. While volume rendering captures overall radiance, surface rendering explicitly decomposes it into diffuse and specular components. This allows the method to handle view-dependent effects of glossy surfaces without misinterpreting highlights as geometric deformations. They also introduce a novel regularization strategy for specular reflections.
  • Stage 2: Learning Direct Lighting Visibility and Indirect Illumination: Building upon the SDF from Stage 1, this stage learns to predict light visibility (crucial for shadows) and indirect illumination. They propose a continuous visibility function derived from the SDF, which is more amenable to network learning than binary visibility. Indirect illumination is represented using Spherical Gaussians (SGs).
  • Stage 3: Recovering Materials and Direct Illumination: In the final stage, the framework reconstructs material properties (specifically, a learnable specular albedo integrated into a Spherical Gaussian BRDF model) and direct illumination. This learnable specular albedo is a significant contribution, allowing for more accurate disentanglement of specular reflections. The direct illumination is also represented using SGs.

Datasets

The paper evaluates its method on several datasets:

  • SK3D [40]: A real-world dataset with complex, glossy objects and lighting.
  • DTU [17]: A real-world dataset, used here for its glossy surface scenes.
  • Shiny [39]: A synthetic dataset with glossy objects.
  • Glossy [25]: A dataset containing both real and synthetic scenes.
  • IndiSG [56]: A dataset specifically designed for evaluating material and lighting reconstruction, providing ground truth for diffuse albedo, roughness, and masks.

Key Insights and Ideas

  • Joint Surface and Volume Rendering is Crucial for Glossy Surfaces: By simultaneously optimizing both, Factored-NeuS avoids the geometric artifacts that arise when highlights are misinterpreted as shape deformations.
  • Explicitly Modeling Specular Albedo Improves Material Decomposition: Introducing a learnable specular albedo parameter allows for a more accurate disentanglement of diffuse and specular components, leading to better material reconstruction.
  • Continuous Visibility Supervision Enhances Shadow and Light Handling: Using a continuous visibility function derived from the SDF, rather than a binary one, makes the learning process more stable and effective.
  • Real-World Data Focus is Key: The authors emphasize that their method's development on real-world data, with surface reconstruction quality as the primary metric, leads to better generalization and robustness compared to methods primarily trained on synthetic data.

Key Results

  • State-of-the-Art Surface Reconstruction: Factored-NeuS achieves the best or second-best performance on glossy scenes in the SK3D and DTU datasets in terms of Chamfer distance.
  • Superior Material and Illumination Reconstruction: Quantitative results on the IndiSG dataset show significant improvements in diffuse albedo, roughness, appearance synthesis, and illumination quality compared to baselines like PhySG, NVDiffrec, and IndiSG.
  • Qualitative Excellence: Visualizations demonstrate that Factored-NeuS can reconstruct detailed glossy surfaces without the geometric artifacts seen in other methods, and it effectively disentangles diffuse and specular components.
  • Robustness to Complex Lighting: The method performs well even under challenging lighting conditions and with highly reflective objects.

Limitations and Future Work

  • Geo-NeuS Advantage with Additional Data: The paper notes that Geo-NeuS can achieve better surface reconstruction on DTU when provided with additional sparse 3D point clouds from Structure-from-Motion (SfM). This suggests that incorporating such data could further improve Factored-NeuS.
  • Fine Structure Decomposition: The authors suggest that a future research direction is to effectively decompose materials for fine structures, implying that current methods might struggle with very intricate details.
  • Potential for Further Improvement with Geo-NeuS Components: The paper mentions that their approach can incorporate components of Geo-NeuS for further surface reconstruction quality improvements, indicating a potential for hybrid approaches.

Summary updated

Abstract

We develop a method that recovers the surface, materials, and illumination of a scene from its posed multi-view images. In contrast to prior work, it does not require any additional data and can handle glossy objects or bright lighting. It is a progressive inverse rendering approach, which consists of three stages. In the first stage, we reconstruct the scene radiance and signed distance function (SDF) with a novel regularization strategy for specular reflections. We propose to explain a pixel color using both surface and volume rendering jointly, which allows for handling complex view-dependent lighting effects for surface reconstruction. In the second stage, we distill light visibility and indirect illumination from the learned SDF and radiance field using learnable mapping functions. Finally, we design a method for estimating the ratio of incoming direct light reflected in a specular manner and use it to reconstruct the materials and direct illumination. Experimental results demonstrate that the proposed method outperforms the current state-of-the-art in recovering surfaces, materials, and lighting without relying on any additional data.
BibTeX
@inproceedings{fan2025factored,
  author = {Yue Fan and Ningjing Fan and Ivan Skorokhodov and Oleg Voynov and Savva Ignatyev and Evgeny Burnaev and Peter Wonka and Yiqun Wang},
  title = {{Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01986},
}
Figure