Most computer vision algorithms see the world through a simplifying lens: they assume every surface is Lambertian. A Lambertian surface is perfectly matte, meaning it reflects light uniformly in all directions. Under this assumption, a physical point on an object retains the exact same color and texture whether you look at it from the left, from the right, or from the center.
This assumption of photometric consistency is the bedrock of 3D reconstruction. It allows stereo matching algorithms to find the same pixel in two different camera views, and it allows self-supervised monocular models to predict depth by warping one video frame into another.
However, the real world is filled with non-Lambertian surfaces: mirrors, glass windows, polished metals, glossy plastics, and bodies of water. When an algorithm encounters these materials, the Lambertian assumption collapses, causing catastrophic failures in geometric estimation.
Why Non-Lambertian Surfaces Break the Math
To understand why non-Lambertian surfaces confuse depth estimation, we have to look at how light behaves when it hits them. Instead of scattering uniformly, light interacting with specular or transparent materials creates two primary failure modes:
- The Multi-Depth Ambiguity: On a transparent surface like a glass window, a single pixel coordinate captures two overlapping signals: the light reflecting off the glass surface and the light transmitting through the glass from the background scene. As Ole Johannsen et al. point out, traditional depth techniques assume each pixel corresponds to a single depth value. When multiple light rays from different depths superimpose, the single-depth assumption is violated.
- Moving Virtual Images: On a highly reflective surface like a mirror, the visual features observed are not physical markings on the surface itself; they are virtual images of the surrounding environment. As the camera moves, these specular highlights shift or vanish. Clement Godard et al. demonstrate that self-supervised monocular depth estimation relies heavily on photometric consistency across viewpoints. Because a reflection's appearance changes depending on the viewing angle, matching algorithms attempt to calculate depth based on the reflected scene rather than the physical surface, leading to massive depth hallucinations or "black holes."
This problem isn't limited to passive cameras. Active sensors like LiDAR and Time-of-Flight (ToF) are equally vulnerable. As noted by Giovanni Pintore et al., active sensors often return erroneous or missing values for mirrors and glass because the emitted light paths either pass straight through or reflect away from the receiver entirely. Consequently, models trained on real-world sensor data inherit these structural gaps, as shown by Rongrong Ji et al..
Lambertian (Matte) Surface Specular (Mirror) Surface
Camera A Camera B Camera A Camera B
\ / \ /
\ / \ /
v v v v
+---------------+ +---------------+
| Physical Point| | Moving Highlight|
| (Same Color) | | (Changes Color) |
+---------------+ +---------------+
Photometric Consistency Photometric Consistency
HOLDS VIOLATED
Leveraging Global Context and Semantics
If local pixel matching fails on shiny and clear surfaces, how can we resolve the geometry? One powerful approach is to shift the burden from local pixel-matching to global semantic understanding.
While a stereo matching algorithm gets trapped trying to find pixel correspondences inside a mirror reflection, a human looker immediately identifies the mirror by its frame, its context, and the implausibility of the reflected room geometry. Computer vision models can mimic this intuition by leveraging global context.
Alex Costanzino highlights a fascinating paradox in the Booster benchmark: while multi-view stereo algorithms fail on specular surfaces because they try to match moving features, monocular depth estimation models can sometimes provide more plausible results. This is because monocular models rely on semantic context to recognize objects and infer their shapes holistically.
To maximize this capability, researchers have turned to architectures designed for global relationships:
- Vision Transformers: By moving away from purely local convolutional operations, models like the one proposed by Kang Chen et al. use the global context-awareness of Transformers to accurately estimate the depth of transparent objects, resolving ambiguities by looking at the entire scene structure.
- Material-Aware Constraints: If a network knows what material it is looking at, it can adapt its geometric assumptions. Tiancheng Zhi et al. demonstrate that integrating material semantic information directly into the network allows the system to account for non-Lambertian surfaces—such as glass windshields—that drastically alter their appearance across different spectral bands.
Turning Reflections into Geometric Constraints
Rather than treating non-Lambertian reflections as noise to be ignored or smoothed over, an alternative paradigm treats them as rich geometric signals. Specular reflections follow strict physical laws (the angle of incidence equals the angle of reflection), meaning the movement of a highlight contains mathematical information about the surface's curvature.
Diego Nehab et al. pioneered this by introducing a "specularity consistency" metric. By observing how a calibrated light source reflects off a mirror-like object across multiple views, they treat the reflections as geometric constraints rather than errors, enabling the triangulation of surface points and normals with extreme precision.
In light-field imaging, where a camera captures both the intensity and direction of incoming light rays, this concept is taken a step further. Because a single pixel coordinate in a light field contains an array of viewpoints, researchers can disentangle the superimposed light layers. Gregoire Nieto et al. propose modeling the plenoptic space directly using linear approximations of ray behaviors, bypassing the need to assign a single static depth value to a refractive or reflective point where depth is fundamentally ill-defined.
Training Networks to Understand Reflection Physics
To teach deep neural networks how to handle non-Lambertian surfaces, researchers need training data that exposes the network to the complex physics of light transport. However, collecting real-world ground truth depth for mirrors and glass is notoriously difficult.
One solution is to use generative models to bridge this gap. Fabio Tosi et al. introduced a framework that uses text-to-image diffusion models to "hallucinate" complex environmental conditions—such as mirrors and heavy reflections—onto simple, geometry-consistent training images. By training on this data, depth estimation networks learn to interpret non-Lambertian surfaces without predicting artifacts like depth holes or flat planes.
+-------------------+ +---------------------+ +-------------------------+
| Geometry-Consistent| --> | Diffusion Model | --> | Robust Depth Network |
| Base Scene (Matte)| | (Add Reflections/ | | Learns to ignore virtual|
+-------------------+ | Glass Surfaces) | | depths & find surface |
+---------------------+ +-------------------------+
Additionally, loss functions can be tailored to explicitly police reflection errors during self-supervised training. For instance, the reflection-aware triplet mining strategy introduced by Wonhyeok Choi et al. uses camera geometry to identify regions where multi-layered reflections violate photometric consistency, preventing these tricky regions from corrupting the network's optimization process.
By combining global semantic context, explicit light-transport physics, and advanced synthetic augmentation, modern depth estimation frameworks are steadily moving past the Lambertian assumption—enabling machines to accurately map a world built out of glass, water, and steel.