Stereo vision mimics human depth perception by comparing two slightly offset viewpoints. By identifying the same point in both a left and a right image, an algorithm can calculate the horizontal shift—or disparity—between them. Because disparity is inversely proportional to depth, finding accurate pixel correspondences allows us to reconstruct a dense 3D map of the scene.
However, this geometric framework relies on a fragile assumption: that every point visible to one camera is also visible to the other. In the real world, this assumption frequently breaks down due to binocular half-occlusion.
The Geometry of the Blind Spot
An occlusion occurs whenever a foreground object blocks a portion of the background. Because the two cameras are separated by a horizontal baseline, they see slightly different configurations of the scene. A region of the background immediately to the right of a foreground object might be visible to the right camera, but completely hidden from the left camera.
Left Camera Right Camera
\ /
\ /
v v
[Foreground Object]
| \
| \ <-- Occluded from Left Camera,
v v Visible to Right Camera
[Background] [Background]
As Geoffrey Egnal et al. established in their foundational comparative work, these half-occluded regions fundamentally lack a true corresponding point in the stereo pair.
When standard stereo matching algorithms attempt to process an image, they typically build a cost volume—a data structure that evaluates the visual similarity of pixel matches across all possible disparities. If an algorithm does not explicitly account for occlusions, it will force a match for every single pixel, even when no true match exists. The optimization framework will simply select the pixel that looks most similar by chance. This behavior introduces significant depth inaccuracies, noise, and artifacts like "flying pixels" or blurred, fattened object boundaries (Yinda Zhang et al.).
The Breakdown of Unsupervised Learning
The occlusion problem becomes even more severe when training deep neural networks without human-labeled ground truth depth. Unsupervised stereo networks rely on a photometric reconstruction loss. The network predicts a disparity map for the left image, uses that map to warp the right image back into the left viewpoint, and measures how closely the reconstructed image matches the original.
As Clement Godard et al. demonstrated, this image reconstruction logic breaks down at occlusion boundaries. Because the occluded background pixels exist in only one camera view, the target image does not contain the visual data needed to reconstruct them. Attempting to force a match in these zones injects corrupt gradients into the network during training, severely degrading the model's ability to resolve sharp object edges (Matan Goldman et al.).
Detecting Occlusions via Left-Right Consistency
To prevent occlusions from corrupting depth maps, a system must first identify where geometry fails. The most elegant way to achieve this is by enforcing left-right consistency.
If a pixel is fully visible to both cameras, computing the disparity from the left image's perspective ($D_L$) and then checking the corresponding pixel's disparity from the right image's perspective ($D_R$) should yield an identical, inverse value. Mathematically, a valid pixel must satisfy:
$$D_L(x, y) = D_R(x - D_L(x, y), y)$$
If the two predicted disparities disagree by more than a small threshold, the pixel is flagged as an occlusion.
Clement Godard et al. integrated this consistency check directly into the loss function of deep networks, allowing models to automatically mask out occluded regions during training. Alex Wong et al. expanded on this by introducing a bilateral cyclic constraint and a spatially-varying adaptive regularization scheme, which dynamically smooths depth estimates around complex boundaries without requiring hand-tuned thresholds.
Historically, stereo matching algorithms treated the reference and target images asymmetrically, optimizing the depth map from only one viewpoint. To resolve the boundary ambiguities this creates, Kuk-Jin Yoon proposed symmetric cost functions within a Markov Random Field (MRF) framework. By evaluating data and discontinuity costs for both viewpoints simultaneously, the model ensures that occlusion boundaries are tightly constrained by the physical geometry of both images.
Resolving Occlusions with Multi-View Redundancy
If a point is hidden from one camera, can we find it by adding more viewpoints?
One approach is to simulate a multi-camera setup during training. Matteo Poggi et al. developed a training framework built on unsupervised trinocular assumptions. By utilizing a virtual three-camera array (left, center, and right) during the learning phase, the model learns to validate correspondences in both directions, significantly reducing border artifacts and occlusion blind spots when operating as a single-image monocular depth estimator at test time.
In dense multi-view or light-field setups, this geometric redundancy becomes a powerful tool. Simon Donné demonstrated a variational multi-image stereo matching approach that models visibility on a per-view basis. Because a scene point occluded in one specific camera pair is highly likely to be visible from a third or fourth camera angle within a dense array, the framework can intelligently aggregate matching costs across unoccluded views, yielding exceptionally sharp depth boundaries.
From Triangulation to Semantic Inpainting
Masking out occlusions prevents errors, but it leaves a fundamental question unanswered: How do we assign a depth value to a pixel we cannot see? Once geometry fails, the system must transition from geometric triangulation to semantic inference.
Zhengming Zhou et al. highlighted a core limitation in traditional binocular architectures: they are highly specialized for geometric matching but lack the flexibility to switch to contextual reasoning when a match is physically missing. By bridging monocular and binocular self-supervised learning, modern networks can use global semantic context—understanding that a certain shape is a continuous wall or a tree trunk—to "inpaint" or hallucinate the correct depth values in unmatchable zones.
This idea of using structural context to guide depth filling also underpins classical methods. For instance, Richard Hartley demonstrated that grouping pixels into homogeneous "superpixels" before handling occlusions prevents false matches from propagating errors across the scene, ensuring that the inpainted depth adheres strictly to visible structural boundaries.
The Challenge of Infinite Depth
A closely related breakdown in correspondence matching occurs at the opposite geometric extreme: infinite depth (such as the sky).
As Azin Jahedi et al. demonstrated through the Spring benchmark, many stereo matching algorithms are built on the rigid assumption that all scene elements exist at a finite, positive distance. When confronted with the sky, the true horizontal shift between views is zero. Without explicit mechanisms to identify and represent zero disparity, models attempt to force a finite depth calculation, leading to massive outlier rates in open-world environments. Just as with occlusions, robust depth estimation requires models to recognize when the standard rules of geometric matching no longer apply.