Abstract
Stereo reconstruction in the presence of specular reflections faces significant challenges. Traditional methods often fail because they rely on the diffuse-world assumption, where a point's appearance is independent of the viewing angle. This paper introduces two least-squares models derived from first principles that explicitly account for reflections. By parameterizing depth, surface orientation, and material properties (up to 5 parameters per pixel), these models generalize diffuse stereo. Despite the high-dimensional state space and non-local interactions between surfaces, the authors demonstrate that efficient inference is possible using a specialized variant of PatchMatch stereo.
The Problem: The Lambertian Assumption
Most stereo reconstruction algorithms rely on the Lambertian assumption: the idea that a point in the world has a constant color regardless of the perspective. However, in real-world scenarios involving shiny cars, glass, or wet roads, this assumption breaks down. Reflections move relative to the observer, leading standard algorithms to identify "false matches" and produce corrupted depth maps. While previous approaches attempted to suppress these errors through regularization or robust statistics, this research proposes modeling the physics of the reflection itself to turn a source of error into a useful signal for reconstruction.
Modeling the Physics of Light
The authors derive two distinct models based on the rendering equation to handle different types of reflective surfaces:
- DNM (Delta-BRDF Model): This model treats surfaces as perfect mirrors. It introduces a "mixing" parameter ($\mu$) to determine the ratio of a pixel's color derived from its own diffuse surface versus a reflected point elsewhere in the scene.
- DNMS (Rough Gloss Model): A more sophisticated model accounting for surface roughness ($\sigma$). Instead of a single reflected point, it integrates light from a cone of directions, simulating the blurry reflections seen on semi-glossy materials.
Inference via PatchMatch
Estimating up to five parameters per pixel (depth, two normal components, mixing, and roughness) is computationally demanding. To make this tractable, the researchers adapted PatchMatch Stereo with several key innovations:
- Continuous Refinement: Utilizing the Ceres Solver for gradient descent on pixel parameters.
- Screen-Space Approximations: To avoid the overhead of full ray-tracing, reflections are calculated by searching within existing image pixels.
- Data-Driven Sampling: A strategy to help the algorithm efficiently "guess" which parts of the scene are being reflected.
Key Results and Insights
Experimental results on synthetic benchmarks (created in Blender) and real-world scenes demonstrate that explicitly modeling reflections significantly improves reconstruction quality.
- Normals are Crucial: The research found that accurate surface normals are the lynchpin of the system. Even minor errors in normal estimation cause reflected rays to point to the wrong scene locations, breaking the model.
- Reflections as Information: By treating reflections as non-local interactions, the model uses the reflected content as an extra geometric constraint, leading to sharper and more physically plausible surface normals.
- Material Recovery: As a byproduct of the depth estimation, the algorithm automatically generates maps of surface shininess and roughness, which can be used for material segmentation.
Limitations and Future Work
While the models show great promise, several limitations remain. The current implementation assumes the reflected object is visible within the camera's field of view ("the off-screen problem") and only accounts for a single light bounce. Furthermore, the DNMS model is approximately 50x slower than standard PatchMatch, suggesting that future work should focus on GPU optimization and the integration of global regularization to further enhance robustness.
Summary updated