TL;DR
This paper extends Neural Radiance Fields (NeRFs) to tackle the multiview photometric stereo problem by explicitly modeling geometry (surface normals) and appearance (BRDF and shadows). This physically-informed approach allows for improved relighting and shape estimation, even when extrapolating to unseen lighting conditions.
What Problem Does This Paper Address?
Traditional multiview photometric stereo (MPS) aims to reconstruct 3D shape and surface reflectance from a set of images of an object taken under varying lighting conditions. While Neural Radiance Fields (NeRFs) have shown remarkable success in novel view synthesis, they are typically "black boxes" that don't explicitly disentangle geometry and material properties. This makes it challenging for them to generalize to new lighting scenarios or to directly extract interpretable geometric information like surface normals. Existing NeRF extensions that try to incorporate lighting often struggle with extrapolation, and directly differentiating density to get normals can be noisy. This paper addresses the need for a NeRF-based approach that can robustly estimate shape and material properties from MPS data, enabling better relighting and extrapolation capabilities.
Methods
The authors propose a novel neural architecture that builds upon the NeRF framework. Instead of a single network predicting color and density, their model decomposes the appearance prediction into several components:
- Geometry Network: This network predicts not only volume density (like in standard NeRF) but also the surface normal direction at each point in space. This explicit normal prediction is crucial for understanding local surface reflectance and is more robust than deriving normals from density gradients.
- BRDF Network: This network learns the Bidirectional Reflectance Distribution Function (BRDF) parameters. It takes the surface normal, light direction, view direction, and a latent code (representing scene properties) as input to predict the reflected radiance. The authors incorporate physical constraints like positivity and reciprocity into the BRDF representation.
- Shadow Network: To account for non-local shadowing effects, a separate network predicts a soft shadow value based on the light direction and the latent code. This avoids computationally expensive ray casting for shadows.
These components are integrated into a volume rendering framework. The model is trained using standard NeRF losses (color and density) but adapted for the MPS setting with multiple light sources and known directions. They also introduce a silhouette loss for objects with known masks to improve geometric accuracy.
Datasets
The primary dataset used is the DiLiGenT-MV dataset. This dataset provides images of 5 objects captured from 21 calibrated viewpoints under 96 different light source directions. The images are in HDR and come with ground truth masks. The authors note that the lighting conditions in the dataset are limited (lights positioned in two rings around the camera), which makes it a good testbed for evaluating extrapolation capabilities.
Key Insights and Ideas
- Explicit Geometry and Appearance Decomposition: By separating the prediction of surface normals, BRDF, and shadows, the model gains a more physically grounded understanding of scene appearance, leading to better generalization.
- Direct Normal Regression: Directly regressing surface normals, rather than deriving them from density gradients, leads to cleaner and more accurate normal maps, which are valuable outputs in themselves for photometric stereo.
- Learned Shadowing: Replacing explicit ray casting with a learned shadow network provides a computationally efficient way to model non-local shadowing effects.
- Extrapolation Power: The physically-informed approach allows the model to extrapolate effectively to lighting and viewing directions not seen during training, a significant improvement over naive NeRF extensions.
Key Results
- Competitive Performance: The proposed model achieves competitive performance on the DiLiGenT-MV benchmark for multiview photometric stereo.
- Accurate Shape Estimation: While numerical errors in normal estimation are higher than some specialized methods, the qualitative results show detailed and smooth shape reconstruction.
- Improved Relighting: The model can synthesize novel views with arbitrary lighting directions, producing realistic shadows and specularities, even for directions far from the training data. Qualitative results demonstrate successful relighting and shadow prediction.
Limitations and Future Work
- Density Representation: The authors acknowledge that the soft density representation of NeRF might limit accuracy for objects with hard surfaces. They suggest that combining their approach with implicit surface representations could yield better results.
- BRDF Constraints: While positivity and reciprocity are enforced, the conservation of energy constraint for BRDFs is left as future work due to the difficulty of imposing integral constraints on neural networks.
- BRDF Input: The current BRDF input uses a simplified angular representation. Exploring more sophisticated representations like Rusinkiewicz angles could improve BRDF modeling.
- Generalization: The authors believe there is potential to improve accuracy by imposing additional constraints on the neural BRDF, such as learning a shared model across objects or using a BRDF database.
Summary updated