Telling Occlusion from Glare: A Cheap, Asymmetry-Aware Left-Right Check for Stereo

Stereo depth estimation matches each pixel in a left image to a pixel in the right image. The oldest reliability test for these matches is the left-right consistency check: match left to right, match the result back from right to left, and discard any pixel that does not return to where it started. The check works well when a background point is visible in only one camera, a situation called half-occlusion. It fails in a less noticed case. When only one camera sees glare, a specular highlight, saturated exposure or heavy noise, the backward search starts from a corrupted pixel, wanders off, and the check deletes a perfectly visible surface as if it were occluded. The check also usually costs a second full network pass, which is hard to afford on the 12-megapixel images where these radiometric problems are worst.

This note proposes two things. The first is to get the backward search almost for free: in rectified stereo with a dot-product correlation volume, the right-to-left volume is an exact re-indexing of the left-to-right one. The second is to replace the binary "returned or not" test with a small verifier that reads why a cycle failed. A true half-occlusion has a geometric signature: the backward match lands on a nearer surface with larger disparity, beside a depth discontinuity, and the pattern follows visibility ordering. One-sided radiometric corruption has a different signature: a flat or multimodal backward cost profile with no occluder geometry behind it. The claim to test is that these two causes of disagreement can be separated by cues already present in a single forward cost volume. If they can, bidirectional filtering stops discarding valid geometry on specular and asymmetric scenes and adds no extra matching cost.

Core Claim

Let $C_L(x, y, d)$ be the correlation between the left feature at $(x,y)$ and the right feature at $(x-d, y)$. Correlation is a dot product, so the right-referenced volume is $$C_R(x', y, d) = C_L(x'+d,\, y,\, d),$$ a shear of the same tensor. The backward winner can therefore be read off the forward volume in $O(HWD)$ indexing time, with no second feature extraction and no second cost construction. This shear is exact only for correlation volumes. Concatenation volumes and direction-specific learned aggregation break it, so the design keeps the check at the raw or lightly aggregated correlation level.

For each left pixel whose cycle fails, the verifier sees three groups of signals and outputs "occluded", "radiometrically corrupted but visible", or "wrong match":

  1. The disparity jump at the backward landing site and whether it points to a nearer surface.
  2. The peak sharpness and modality of the forward and backward cost profiles.
  3. A cheap per-pixel degradation estimate for each view, for example local saturation, gradient energy and noise level.

Pixels labelled "visible but corrupted" keep their forward estimate, which is down-weighted rather than deleted. Pixels labelled "occluded" are removed or filled as usual. The new element is the explicit separation of occlusion from one-sided corruption. The cheap shear by itself is a known stereo identity.

Relation to Prior Work

Several papers motivate the gap. Classical work showed that left-right disagreement is an accurate stand-in for half-occlusion: Combining Stereo and Monocular Information to Compute Dense Depth Maps that Preserve Depth Discontinuities and Detecting Binocular Half-Occlusions: Empirical Comparisons of Four Approaches compared detectors and found left-right checks competitive. Analysis of KITTI Data for Stereo Analysis with Stereo Confidence Measures found the check among the most dependable confidence measures in driving scenes. All three assume symmetric observability, which is exactly the assumption this proposal relaxes.

Open Challenges in Deep Stereo: the Booster Dataset documents the regime where that assumption breaks. It reports errors above 80% on transparent and specular objects and bad-2 above 40% over all pixels at full resolution. Unsupervised Deep Asymmetric Stereo Matching with Spatially-Adaptive Self-Similarity (SASS) handles one-sided noise and blur by matching in a learned self-similarity space and uses bidirectional constraints in training. It tries to make features symmetric. It does not diagnose disagreement, and it was evaluated at downsampled resolution.

The closest prior method is CBMV: A Coalesced Bidirectional Matching Volume for Disparity Estimation. CBMV feeds left-to-right and right-to-left confidences from four radiometrically robust costs into a random forest, so bidirectional signals become features. It builds separate volumes per direction and per cost, takes about 162 seconds per KITTI pair on a CPU, and never asks why the two directions disagree. This proposal keeps CBMV's lesson that bidirectional signals are informative inputs. It drops the duplicated volumes and adds an explicit occlusion-versus-corruption label.

Two other bidirectional methods differ in setting and cost. Cross-View Completion Models are Zero-shot Correspondence Estimators (ZeroCo) enforces reciprocity by running a ViT decoder twice. Matcher applies reciprocal filtering to coarse DINOv2 patches. Both pay the doubled cost the shear avoids, and both use patch-level rather than pixel-level disparity.

Prior work also supplies the efficient backbones the verifier must plug into. Parameterized Cost Volume for Stereo Matching (PCVNet) models disparity as a four-component Gaussian mixture. It is unidirectional and handles occlusion through a variance heuristic. Decoupling Bidirectional Geometric Representations of 4D cost volume with 2D convolution (DBStereo) aggregates the volume with 2D and 1x1 convolutions. Its "bidirectional" refers to spatial versus disparity axes, not left-right checking. Both build correlation volumes, so the shear applies before their aggregation.

Research Program

The first step is an analysis study that tests whether the separation is real before any method is built. On SceneFlow, occlusion masks are computed from the ground-truth left and right disparities. Controlled one-sided corruptions are then applied to the right image: SASS-style resolution and noise asymmetry, synthetic saturated highlights, and exposure shifts. Every failed cycle on a known-visible pixel is a false rejection.

The analysis measures how far simple, hand-defined cues already separate the two failure types:

  1. The sign and size of the landing-site disparity jump.
  2. The ratio of the backward cost peak to the second-best peak.
  3. The degradation estimate.

The measure is an ROC curve for occlusion versus corrupted-visible. If hand cues alone reach high separation, the verifier can stay tiny. If they do not, the idea is in trouble and the note will say so.

The second step builds the verifier as a small convolutional head over the sheared volume and the cue maps. It is attached to two backbones, PCVNet and DBStereo, using the PCVNet codebase. It is trained on SceneFlow with randomized one-sided corruption, so corruption labels come for free.

At high resolution, the shear is applied at the coarse correlation level where the backbone already builds its volume. Verifier outputs are upsampled with the disparity. Memory therefore stays at the backbone's $O(HWD)$ rather than doubling.

The third step uses the verifier's "corrupted but visible" label as training signal, not only as a post-filter. The disparity loss on those pixels is down-weighted, and the backbone is fine-tuned on Booster. This tests whether better diagnosis also improves the depth estimate, as opposed to only improving the mask.

Baselines

The naive reference point is the backbone with no filtering at all. Its full-density error on Booster and on corrupted KITTI fixes the floor.

The standard competitor is the classical hard left-right check run as post-processing on two independent network passes. This is what practitioners do today. The same check is also run on the free sheared volume, which isolates the shear's effect on runtime from any effect on accuracy. Both use a tolerance of one pixel, chosen on the same validation split as the verifier.

The strongest existing system on the non-Lambertian question is PCVNet. Parameterized Cost Volume for Stereo Matching reports a Booster RMSE of 15.42 against RAFT-Stereo's 17.39. The Booster paper reports that fine-tuned RAFT-Stereo reaches an MAE of 7.56 pixels. Beating PCVNet means lower RMSE on Booster at equal density, or the same RMSE at clearly higher density. The comparison must be at matched density, because a filter can trivially lower error by deleting pixels.

For one-sided asymmetry, SASS is the strongest competitor. On simulated KITTI 2015 asymmetry it reports EPE 2.183 under 4x right-image downsampling and 2.667 under noise with σ=0.15. The verifier will be evaluated under exactly that protocol.

CBMV is the strongest prior method that uses bidirectional confidences as features. It reports an 11.1% average error on Middlebury 2014. It will be run on Middlebury and KITTI to compare accuracy and runtime.

Two ablations isolate the key ingredient:

  1. A verifier given identical inputs except the backward and landing-site cues, which is effectively a learned unidirectional confidence head.
  2. A verifier with only two output classes, consistent and inconsistent.

All learned components receive the same training data, corruption augmentation, GPU budget and hyperparameter search budget.

Why not just…?

Why not just run the standard left-right check after two passes? It has two problems. It doubles runtime and memory, and it has no way to tell glare from occlusion, so on Booster's specular regions it should delete visible surfaces. The shear removes the first problem with no learning at all. The false-rejection rate measured on known-visible pixels under one-sided corruption tests the second directly. If that rate is low for the plain check, the verifier is unnecessary and the note's claim fails.

Why not just make the features invariant, with Census or SASS, so the two directions agree again? This is the most serious threat. If symmetric features remove one-sided degradation, a plain check would stop misfiring. The evidence against it is partial. Evaluation of Cost Functions for Stereo Matching shows that ordinal transforms assume monotonic intensity changes, which specular reflection violates, and Booster's errors on mirrors and glass suggest learned features do not fully recover either. The experiment is direct: SASS-style and Census-augmented features are combined with the plain check. If false rejections on Booster's specular masks drop to near the occlusion-only level, the proposal reduces to "use invariant features", and the note will report that.

Why not just train a confidence network on the forward volume? A learned unidirectional confidence head can down-weight bad pixels without any backward search. Its weakness is that it cannot see the occluder geometry at the backward landing site, which is the cue that marks occlusion. Ablation 1 above is exactly this baseline. If it matches the full verifier at occlusion-versus-corruption separation, the bidirectional component adds nothing.

Proposed Experiments

Training uses SceneFlow with one-sided corruption augmentation. Evaluation covers four settings:

  1. Booster at full and quarter resolution, reporting RMSE, MAE and bad-2, broken out by Booster's material masks. The results file does not establish whether Booster supplies right-view disparities. If it does not, occlusion labels will be derived by warping the left ground truth.
  2. KITTI 2015 under the SASS asymmetry protocol, using its non-occluded versus all-pixel split.
  3. Middlebury 2014.
  4. ETH3D, for generalization.

Every accuracy number is paired with density. The primary diagnostic metrics are the false-rejection rate on visible pixels and the occlusion-detection AUC. Runtime and peak memory are reported on one GPU for the plain check, the shear check, and the verifier.

The decisive checks are three. First, at matched density, the verifier must lower Booster error on specular and transparent pixels relative to the plain check, without raising error at true occlusions. Second, the learned unidirectional confidence ablation must fall measurably short, which shows the backward geometry is doing the work. Third, the added runtime over the bare backbone must be a small fraction of a second pass. Failing the first check kills the idea. Failing only the second reduces it to the cheap-shear engineering result, which is still useful but not the claim made here.