TL;DR
This paper presents a novel method to automatically reconstruct spatially varying Bidirectional Reflectance Distribution Functions (SVBRDFs) for 3D objects captured in photogrammetry studios. By jointly optimizing the object's geometry, lighting configuration, and material properties from standard RGB images and an HDR environment map, the approach significantly reduces the need for manual, time-consuming texture painting, enabling photorealistic rendering of captured assets.
What Problem Does This Paper Address?
Photogrammetry studios are excellent at capturing high-quality 3D geometry of objects. However, creating realistic renderings typically requires detailed material properties (BRDFs) that are often manually painted by artists. This process is labor-intensive and expensive. While specialized "light stage" setups can automate BRDF acquisition, they are prohibitively costly and inflexible. This paper aims to bridge this gap by enabling automatic SVBRDF reconstruction within more accessible photogrammetry studio setups, requiring only a slight extension to the standard capture pipeline.
Methods
The core of the proposed method is a joint optimization framework that leverages existing photogrammetry outputs (3D geometry, camera parameters) and a captured HDR environment map.
- Input: High-quality 3D geometry, precise camera parameters (intrinsic and extrinsic), and a roughly aligned HDR environment map of the studio lighting.
- Illumination Representation: The HDR environment map is represented as a set of directional lights, reduced to a manageable number (e.g., 256) using a median cut approach. Crucially, the method also estimates the position and orientation of this environment map, as well as the distance of individual lights, acknowledging that lights are not infinitely far away.
- BRDF Model: The Disney BRDF model is used due to its expressiveness and widespread use in production.
- Optimization: The problem is formulated as minimizing photometric error between rendered and input images. This is achieved through an alternating optimization strategy:
- Light and Reflectance Estimation: Given the diffuse albedo, the method optimizes for the SVBRDF parameters and the light setup (environment map pose, light distances). This step uses high-luminosity samples that are more sensitive to specular reflections.
- Albedo Estimation: Given the estimated light setup, the method optimizes for the diffuse albedo. This step uses low-luminosity samples.
- Regularization: To improve results and handle texture boundaries, a local smoothness regularizer is applied to the BRDF parameters. A similarity term is also introduced to encourage neighboring light sources in the environment map to have similar distances, improving the stability of light reconstruction.
Datasets
The authors evaluate their method on both synthetic and real-world datasets.
- Synthetic Data: Three datasets are generated with varying complexity, including diffuse reflectance and a complex Disney BRDF. These datasets are rendered from camera positions mimicking a photogrammetry studio setup.
- Real-World Data: Four real-world scenes are captured: an "UPPER BODY," a "BUST," a "SHOE," and a "VASE." These objects represent different challenges in terms of geometry and reflectance complexity. For evaluation, the data is split into "fit" (for reconstruction) and "unseen" (for generalization) sets.
Key Insights and Ideas
- Beyond Directional Lights: The paper convincingly demonstrates that treating lights as infinitely distant (directional) is a significant simplification that leads to higher photometric errors compared to modeling them as point lights with estimated positions and distances.
- Joint Optimization is Key: Separately estimating geometry, lighting, and materials is insufficient. The tight coupling between these elements necessitates a joint optimization approach for accurate results.
- Low-Luminosity for Albedo, High-Luminosity for Specular: The insight that low-luminosity image regions primarily contain diffuse albedo information, while high-luminosity regions reveal specular highlights and thus light positions, is crucial for the alternating optimization strategy.
- Light Similarity Regularization: The introduction of a similarity term for light distances significantly improves the robustness of light reconstruction, especially when the number of estimated lights differs from the ground truth.
Key Results
- Photometric Accuracy: The method achieves low photometric RMSE on real-world data, ranging from 1.6% to 2.3% when using all input images. For the VASE scene, the error is reduced to 1.65%.
- Light Reconstruction Accuracy: On synthetic data, the method achieves accurate light position reconstruction, with an average error of around 10.7cm for complex BRDFs.
- BRDF Reconstruction: The estimated SVBRDF parameters closely match ground truth in synthetic experiments, with an average relative deviation of 2.1%.
- Runtime: The GPU-accelerated implementation is efficient, running in approximately 3.5 hours for a complex scene, a significant improvement over naive implementations.
Limitations and Future Work
- Geometry and Camera Accuracy: The method is sensitive to inaccuracies in the input geometry and camera alignment, especially for highly specular areas or high-frequency albedo details.
- Anisotropic Materials: The current Disney BRDF model does not fully capture anisotropic materials, and future work could extend the optimization to handle these.
- Self-Shadowing and Indirect Illumination: The current image formation model neglects self-shadowing and indirect illumination. Integrating these effects, potentially with deep learning advancements, is a promising future direction.
- Assisted Reconstruction: The authors suggest that their method could potentially assist stereo-based 3D reconstruction in improving surface normals for challenging, glossy areas.
Summary updated