In computer vision, depth estimation and 3D reconstruction often rely on a simple but powerful assumption: a point in the real world should look the same regardless of which camera view it is seen from. This principle, known as photometric consistency, allows models to learn depth without human labels by "warping" one image into the perspective of another and measuring the error.
However, this assumption breaks down at the edges of objects. When a foreground object blocks the view of the background in one camera but not the other, we encounter an occlusion. If a model tries to enforce consistency in these occluded regions, it receives "noisy" gradients, leading to artifacts like "foreground fattening" (where objects appear unnaturally thick) or "flying pixels" (depth values that bleed into empty space).
To solve this, researchers have developed a suite of mechanisms to identify and mask these occluded regions, ensuring that geometric consistency is only enforced where it is physically valid.
The Geometry of Visibility
The most direct way to handle occlusions is to predict them using the geometry of the camera setup. In stereo vision, Yinda Zhang and colleagues in ActiveStereoNet popularized the use of a left-right consistency check. By predicting depth from both the left and right camera perspectives, the system can identify pixels that don't map back to their original starting point. If the "round trip" from the left image to the right and back again fails, the pixel is likely occluded and should be masked out of the training loss.
This concept extends beyond simple stereo. In DeProCams, Bingyao Huang uses epipolar geometry to generate differentiable occlusion masks for projector-camera systems. By searching along 1D epipolar lines, the model can identify shadows and occlusions where the projector's light cannot reach the camera's view, preventing the system from trying to reconstruct geometry in the dark.
Differentiable Z-Buffering and Depth Ordering
In monocular video or Neural Radiance Fields (NeRFs), we don't have a fixed second camera. Instead, we have a sequence of frames. To handle occlusions here, models must reason about which surfaces are "in front."
Ariel Gordon and others introduced a differentiable Z-buffer approach in Depth From Videos in the Wild. When warping a source frame to a target frame, the model checks if multiple source pixels map to the same target location. By comparing their estimated depths, the model can "hide" the background pixels that are obscured by the foreground.
Similarly, GeCoNeRF by Min-Seop Kwak uses the model's own depth estimates to create "pseudo ground truths." By comparing the rendered depth of a target view with the reprojected depth from a source view, the system can identify discrepancies. If the depths don't match, it indicates an occlusion, and the dynamic visibility mask excludes that region from the optimization.
Solving the Edge Fattening Problem
One of the most persistent issues in depth estimation is the "bleeding" of depth values across object boundaries. Shengjie Zhu and Xiaoming Liu demonstrate in The Edge of Depth that explicit occlusion masks are essential for maintaining sharp boundaries. Without them, the photometric loss tries to minimize error by averaging the depth of the foreground and background, resulting in blurry, "fattened" edges.
In the context of light fields, Junmo Kim proposed a strategy of successively excluding foregrounds. By identifying and masking foreground objects first, the algorithm can "see behind" them to estimate background depth using only the rays that are not obstructed. This prevents the color of the foreground from contaminating the depth calculation of the background.
Leveraging Visibility Asymmetry
Recent work has moved from simply ignoring occlusions to exploiting them. In Self-Supervised Stereo Matching with Multi-Baseline Contrastive Learning, Eryun Liu and Jingyun Fu use a teacher-student framework where the teacher is given a different viewpoint than the student.
This creates a visibility asymmetry: the teacher can "see" into the regions that are occluded from the student's perspective. Instead of the student guessing in the dark, the teacher provides reliable supervision for those tricky occluded areas. This shift from passive masking to active cross-view supervision represents the current frontier in making 3D vision systems robust to the complexities of the real world.