TL;DR
Factored-NeuS is a three-stage inverse rendering framework that reconstructs high-quality geometry, materials, and lighting from multi-view images. By jointly optimizing volume and surface rendering and introducing a learnable specular albedo, it effectively eliminates the "bumpy" geometric artifacts that typically plague the reconstruction of glossy objects.
What Problem Does This Paper Address?
Reconstructing 3D scenes from images is a cornerstone of computer vision, but glossy surfaces remain a major hurdle. Standard implicit surface methods (like NeuS) often fail because specular highlights change depending on where you stand. To explain these view-dependent color shifts, neural networks often "cheat" by creating fake geometric dents or bumps—essentially trying to hide the highlight inside a concave deformation.
Furthermore, disentangling a pixel's color into its base material (diffuse albedo), its shininess (specular component), and the environmental lighting is an under-determined problem. Existing methods often "leak" specular highlights into the diffuse component, resulting in inaccurate material maps that look like they have highlights painted onto them.
Methods
Factored-NeuS employs a progressive three-stage pipeline to factorize the scene:
- Stage 1: Geometry Reconstruction. The authors propose a joint rendering strategy. They use volume rendering (standard in NeRF-style models) alongside surface rendering. Crucially, they decompose the radiance into diffuse and specular components during this stage. By forcing the volume-rendered color and the surface-rendered color to match, they regularize the Signed Distance Function (SDF), preventing the formation of geometric artifacts caused by highlights.
- Stage 2: Visibility and Indirect Light. Using the frozen SDF from Stage 1, the model learns to predict light visibility and indirect illumination. Instead of binary "shadow or no shadow" ray-tracing, they use a continuous visibility function derived from the SDF weights, which is much easier for neural networks to optimize.
- Stage 3: Material and Direct Light. The final stage recovers the BRDF (using the Ward model) and direct environment lighting (represented by Spherical Gaussians). A key novelty here is the specular albedo network ($M_{sa}$), which learns a per-point ratio of specular reflection, allowing for much more accurate material decomposition on complex real-world surfaces.
Datasets
The authors evaluated their method on a mix of real and synthetic datasets:
- SK3D: A real-world dataset featuring highly reflective objects with complex lighting.
- DTU: A standard multi-view benchmark for 3D reconstruction.
- Shiny: A synthetic dataset specifically designed to test glossy surface reconstruction.
- IndiSG: A dataset with ground-truth material maps (albedo, roughness) used to evaluate decomposition accuracy.
- Glossy: A dataset containing both real and synthetic scenes.
Key Insights and Ideas
- Joint Rendering as Regularization: The most significant "aha moment" is that surface rendering handles specular reflections better, while volume rendering is more robust for general shape. Using them jointly forces the SDF to remain smooth even in the presence of intense highlights.
- Continuous Visibility: Moving from discrete ray-traced visibility to a continuous integral representation based on SDF weights makes the visibility function differentiable and much more stable for network training.
- Learnable Specular Albedo: By not assuming a uniform specular response across the object, the $M_{sa}$ network allows the model to distinguish between parts of an object that are truly glossy and those that are more matte, leading to cleaner albedo maps.
Key Results
Factored-NeuS achieves state-of-the-art results across multiple benchmarks:
- Geometry: On the glossy scenes of the SK3D dataset, it achieved a mean Chamfer distance of 1.55, significantly outperforming NeuS (2.31) and NeRO (3.65).
- Material Decomposition: On the IndiSG dataset, it reached a mean PSNR for diffuse albedo of 21.21, compared to IndiSG's 22.47 and NVDiffrec's 13.56.
- Qualitative: As shown in the figures, the method produces clean, smooth surfaces for objects like metal pots and ceramic jugs where previous methods produced "melted" or dented geometry.
Limitations and Future Work
Despite its strengths, the method has some limitations:
- Fine Structures: It still struggles to reconstruct very small, intricate details like the tiny nails on the back of a chair or fine textures on a pillow.
- Fresnel Effects: While increasing the Fresnel value helps with chrome-like materials, it can sometimes lead to a loss of fine texture details in the albedo.
- Future Directions: The authors suggest that future work should focus on better decomposition strategies for fine-grained structures and more adaptive handling of Fresnel reflections.
Summary updated