Introduction
Reconstructing the full appearance of a scene—encompassing both its geometric structure and its spatially-varying bidirectional reflectance distribution function (svBRDF)—remains a core challenge in computer vision. Traditional methods often rely on controlled environments like light stages or are restricted to small, isolated objects. Handheld reconstruction introduces significant ambiguities: distinguishing whether a pixel's brightness stems from surface orientation (geometry) or material shininess (reflectance) is a fundamentally ill-posed problem. Furthermore, the memory requirements for high-resolution 3D volumes often preclude the reconstruction of large-scale environments. This paper presents a scalable framework that jointly optimizes geometry, poses, and materials using a novel distributed approach.
Methodology
The core of the proposed system is a shift from a monolithic 3D representation to a collection of 2.5D keyframe parameter maps. Each keyframe stores depth, normals, and material properties, including diffuse albedo, specular albedo, and roughness. This local representation ensures that memory usage remains constant relative to the scene's total volume, enabling the capture of multi-meter environments.
Distributed Optimization and Consistency
To ensure global consistency across these local maps, the authors employ a decentralized block coordinate descent algorithm. Keyframes are optimized in parallel, with periodic synchronization steps that project information between overlapping views. A multi-view consistency regularizer is introduced to penalize discrepancies between these views. This prevents the optimization from "cheating"—for example, by incorrectly warping a surface normal to explain away a specular highlight that it couldn't otherwise model.
Differentiable Rendering
The framework utilizes a differentiable rendering pipeline based on a modified Cook-Torrance reflectance model. This allows the system to use standard gradient-based solvers to minimize the photometric error between rendered images and actual sensor observations. By optimizing camera poses and material parameters simultaneously, the system resolves ambiguities more effectively than sequential methods; better poses lead to more accurate normals, which in turn yield superior material estimates.
Experimental Setup
The researchers evaluated their method using a custom handheld rig equipped with a high-resolution RGB camera, an active depth sensor, and 12 high-power LED point light sources. The system was tested on various scales:
- Individual Objects: Challenging items such as a 'Gnome', 'Teapot', and 'Globe'.
- Large-Scale Scenes: Multi-object setups and indoor environments like a 'Sofa' and an 'Office' spanning up to 2x3 meters.
- Synthetic Data: Used for controlled quantitative validation of the optimization framework.
Key Results
Quantitative evaluations demonstrate a significant leap over existing 2.5D baselines. For instance, on the 'Duck' dataset, the proposed method reduced photometric test error (a measure of how well the model predicts unseen views) from 31.212 to 10.366.
Qualitatively, the method successfully decouples "baked-in" lighting from true material properties. The resulting 3D meshes feature sharp geometric details and realistic, view-dependent specular highlights that remain consistent as the virtual camera moves. Most notably, the system scales effectively to multi-meter scenes, maintaining a resolution of $\leq 2mm$ in an office environment, demonstrating its capability beyond simple object-level captures.
Limitations and Future Work
While the framework is highly scalable, it currently refines existing geometry rather than generating it from scratch; it cannot "fill in" large holes where the initial depth sensor failed. Additionally, the current model assumes single-bounce light transport, which can lead to small errors in diffuse albedo maps due to ignored global illumination (light bouncing off walls). Future research will focus on extending the model to handle ambient environmental lighting and more complex material properties, such as anisotropy in brushed metals or subsurface scattering in materials like wax.
Summary updated