Specular Object Reconstruction Behind Frosted Glass by Differentiable Rendering

Takafumi Iwaguchi, Hiroyuki Kubo, Hiroshi KawasakiWACV 2024

TL;DR

This paper introduces a novel differentiable rendering approach to reconstruct the 3D shape of objects hidden behind frosted glass by analyzing specular reflections. By leveraging multiple observations under varying light conditions, their method effectively disentangles complex light transport and accurately recovers object geometry, even in challenging scenarios with overlapping reflections.

What Problem Does This Paper Address?

Imaging and reconstructing objects obscured by diffusers like frosted glass or fogged windows is a common but challenging problem. Traditional computer vision techniques struggle because the diffuser blurs the scene, making direct observation impossible. While specular reflections can offer sharper cues, their complex interaction with light and surfaces makes them difficult to interpret. Existing methods for "shape from caustics" often focus on designing surfaces to create specific patterns, rather than reconstructing existing geometry. This paper tackles the problem of recovering the 3D shape of specular objects hidden behind a diffuser by analyzing the patterns of light reflected off these objects and projected onto the diffuser.

Methods

The core of the proposed solution is a rasterizer-based differentiable renderer. This renderer is designed to efficiently model the light transport path: light source -> reflector (object) -> diffuser -> camera.

Here's a breakdown of their approach:

  • Differentiable Rendering: Unlike traditional rendering, a differentiable renderer allows for backpropagation of errors. This means that by comparing the rendered image to a real captured image, the system can compute gradients and update scene parameters (like object shape and light positions) to minimize the difference.
  • Specular Reflection Focus: The method specifically exploits specular reflections, which are stronger and carry higher frequency information than diffuse reflections. This allows for sharper patterns on the diffuser, even from smooth surfaces like paraboloids.
  • Light Transport Modeling: They model the light transport using a two-pass approach. First, they calculate the irradiance on the mirror surface and then project it onto the diffuser. This is done by considering the conservation of light flux between the mirror and the diffuser, accounting for angles and areas.
  • Mesh Representation: Objects are represented as polygon meshes. The renderer then simulates how light reflects off these mesh surfaces and projects onto the diffuser.
  • Optimization: The system optimizes scene parameters (light positions, mirror depth, surface normals, and reflectance maps) by minimizing a data-fidelity loss (difference between rendered and observed images) and a total variation regularization term for normals to ensure smoothness.
  • Multiple Observations: Crucially, the method uses multiple observations captured with different light source positions. This is essential to resolve ambiguities inherent in single observations, as different light angles reveal different aspects of the object's shape and position.

Datasets

The paper describes experiments conducted in both simulated and real-world environments.

  • Simulated Data: Used for initial testing and evaluation of the core reconstruction capabilities. Ground-truth observation images were generated using the physics-based renderer LuxCore.
  • Real-world Data: Collected using a specific experimental setup involving a diffuser, a target object (mirrors or other shapes), a laser light source, and a camera. The process involved capturing images with and without the object to isolate the reflected light patterns.

No new benchmark datasets were introduced; rather, the authors demonstrate their method's effectiveness on synthesized and captured data.

Key Insights and Ideas

  • Specular Reflections as a Key to Hidden Geometry: The paper strongly emphasizes that specular reflections, often a nuisance in other imaging tasks, are the crucial signal for reconstructing objects behind diffusers.
  • Differentiable Rendering for Inverse Problems: Applying differentiable rendering to this inverse problem (recovering scene parameters from observed images) is a powerful paradigm shift, enabling efficient optimization.
  • Multi-view Geometry for Ambiguity Resolution: The success hinges on using multiple light source positions. This is a classic computer vision principle applied here to disentangle complex light interactions and resolve inherent ambiguities in specular reflection patterns.
  • Efficient Caustics Rendering: The proposed method can efficiently render smooth caustics, which is a known challenge for many existing physics-based renderers, especially in optimization contexts.

Key Results

  • Accurate Shape and Depth Reconstruction: The method successfully reconstructed the depth and shape of flat mirrors, even when their reflections overlapped in the observed images. For simulated flat mirrors, depth estimation errors were minimal (e.g., ~0.1-0.4 units).
  • Normal Estimation for Complex Objects: The system demonstrated the ability to estimate surface normals for various object shapes, including concave, convex, and curved surfaces, in simulation. Mean angular errors (MAE) for normal estimation were reported to be as low as 0.487 in simulation.
  • Real-world Validation: Experiments with real objects (letters 'C' and 'V', a pot lid, a dish) showed successful reconstruction of their geometry and reflectance, validating the method's practical applicability.
  • Dependence on Observation Number: The results showed that while a few observations (around 4) are sufficient for noiseless data, more observations can sometimes lead to increased error due to oversmoothing from regularization, highlighting a trade-off. Noise in observations degrades accuracy, but moderate noise can help avoid local optima.

Limitations and Future Work

  • Limited Bounces: The current method cannot render scenes with more than two light bounces. This is a significant limitation for scenes with complex indirect illumination.
  • Future Work: The authors explicitly state that resolving the multi-bounce rendering issue will be a focus for future research. Additionally, further work could explore handling more complex diffuser properties or a wider range of object materials beyond specular.

Summary updated

BibTeX
@inproceedings{iwaguchi2024specular,
  author = {Takafumi Iwaguchi and Hiroyuki Kubo and Hiroshi Kawasaki},
  title = {{Specular Object Reconstruction Behind Frosted Glass by Differentiable Rendering}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure