Abstract
While Neural Radiance Fields (NeRF) have transformed 3D reconstruction, they often struggle with textureless surfaces and complex light transport. Active sensors like the Intel RealSense mitigate this by projecting infrared patterns, but their internal depth-processing algorithms frequently fail in the presence of inter-reflections or translucency. This paper introduces a volumetric image formation model that directly integrates raw structured light measurements into a neural framework. By modeling the physics of projector illumination, the proposed method achieves high-fidelity geometry, detailed surface normals, and a complete decomposition of scene appearance from minimal viewpoints.
Introduction
Traditional 3D reconstruction relies heavily on finding correspondences between images, a task that becomes nearly impossible in "textureless" regions such as plain white walls. Active depth cameras address this by projecting a known pattern (e.g., a grid of dots) to create artificial texture. However, most existing neural reconstruction methods treat the processed depth maps from these sensors as ground truth. This is problematic because the internal heuristics of these cameras often break down when light bounces multiple times or passes through semi-transparent materials.
Researchers from Carnegie Mellon, Meta Reality Labs, and Brown University argue that instead of relying on these potentially erroneous "processed" depth maps, the raw structured light images should be explicitly modeled within the optimization process.
Methodology: Modeling Sensor Physics
The core of this approach is a physically-based neural volume rendering model that accounts for the distinct components of light transport. The system captures both "ambient-only" and "structured light" images by strobing the projector. The neural network is then optimized to minimize the difference between these captured images and those synthesized by the volumetric model.
The model decomposes light into three primary components:
- Ambient ($L^{ambient}$): Environmental light captured when the projector is inactive.
- Direct ($L^{direct}$): Light traveling from the projector to a surface and reflecting once back to the camera. This component incorporates the known projector pattern, inverse square light fall-off, and surface orientation.
- Indirect ($L^{indirect}$): Light that undergoes multiple bounces or subsurface scattering before reaching the sensor.
By tying predicted surface normals to the gradient of the volume density, the framework ensures that geometry and shading remain physically consistent.
Key Insights and Results
The shift from processed depth to raw data yields several significant advantages:
- Radiometric Cues for Normals: The appearance of projected dots—varying in brightness based on surface tilt—provides "radiometric cues" that allow the model to recover incredibly detailed surface normals. In synthetic tests, the method reduced Mean Angular Error (MAE) for normals from 24.34° (standard NeRF) to just 2.84°.
- Handling Translucency: Unlike standard 2D depth maps, the volumetric model can represent multiple surfaces along a single ray. This allows it to successfully reconstruct objects inside translucent containers where traditional depth engines fail.
- Intrinsic Decomposition: The framework naturally separates the scene into direct, indirect, and ambient components, facilitating advanced photo editing and relighting.
In few-view scenarios (e.g., 2 to 4 views), this method significantly outperforms standard NeRF and depth-supervised variants, providing superior geometry even in challenging real-world scenes like woodshops or complex sculptures.
Limitations and Future Work
Despite its robustness, the method faces challenges with extremely specular (mirror-like) surfaces and complex refractive objects. Additionally, in very bright outdoor environments, the low signal-to-noise ratio of the projector pattern can hinder performance. Future work may involve more complex shadow modeling and the integration of multi-camera data to further enhance robustness in diverse lighting conditions.
Summary updated