Stereo Matchers as Judges of Hallucinated Disocclusions

When a single image or video is converted into a stereo pair, or when any camera is moved to a new viewpoint, some background surfaces become visible that were hidden behind foreground objects. These newly exposed regions are called disocclusions. Their pixels have no source, so a generative model has to invent them. Current systems judge that invented content by how it looks, using pixel error, perceptual similarity, or human preference. The content is not seen alone, though. It is seen by two eyes, and the viewer's visual system solves a stereo correspondence problem on it: it searches the other eye's image, along the same horizontal line, for each point's partner. A correct disocclusion has a specific signature under that search. It should have no partner in the other view, because in reality the other eye sees the foreground object at that spot. An inpainter that copies nearby background texture, or the foreground itself, into the hole can create a false partner. The brain then reads a wrong depth, or the two eyes see conflicting images (binocular rivalry).

We propose to use a trained, occlusion-aware stereo matching network as a frozen judge of generated stereo views, both as a training signal and as a metric. The judge asks two questions of the pair (left image, generated right image). First, does it recover the intended disparity (the horizontal shift between the two views, which encodes depth) on visible pixels? Second, does it classify exactly the disoccluded pixels as having no match in the other view? The new ingredient is the second question. It constrains the texture inside the hole, and neither a disparity loss nor an image loss can impose that constraint.

Core Claim

The disocclusion mask is not just a region where a generator must produce plausible pixels. It is a geometric prediction about what a stereo matcher will fail to find. A generated right view is stereo-consistent only if an independent matcher, run with that view as reference, produces an occlusion map that coincides with the analytically known disocclusion mask. A generator trained to satisfy this condition should produce holes that are more comfortable to view binocularly and less prone to false depth. This should hold even at equal or slightly lower pixel fidelity. The claim is falsifiable in two ways. The criterion could fail to track human judgments of rivalry, or existing self-supervised inpainters could already satisfy it.

Three bodies of prior work motivate this. Cost-volume analysis (CNN-Based Cost Volume Analysis as Confidence Measure for Dense Matching) shows that the shape of matching costs across candidate disparities reliably distinguishes real matches from occlusions. A cost volume stores, for each pixel, how well it matches each candidate shifted pixel in the other view. Local Detection of Stereo Occlusion Boundaries. goes further: a small network reading only the cost volume finds occlusion boundaries with an ODS F-score of 0.61 on Middlebury, against 0.37 for PSM-Net. It notes that such boundaries are exactly where matching fails, with error above 50% in regions seen by one eye only. Global Occlusion-Aware Transformer for Robust Stereo Matching (GOAT) supplies the kind of judge we need. It estimates disparity and occlusion with separate attention paths, and in flagged occluded regions it fills depth from global context rather than local costs. This design point matters later.

On the disocclusion side, three systems treat depth and warping as a fixed preprocessing step and evaluate appearance only. Geometric Reciprocity: Unlocking Self-Supervision for Stereoscopic Video Generation (GRT) proves that the target-view disocclusion mask equals the mask of pixels lost when warping back to the source. It uses this to train inpainters on monocular video. M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversion refines warped video in one pass of a video diffusion model, using attention restricted to hole pixels. Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversion shows that pixel metrics reward simply copying the left view (the "identity mapping trap"). It proposes the Stereo IoU (SIoU) edge metric, which correlates with human judgment at 0.84 Spearman, versus 0.26 for RMSE. None of the three ever asks a matcher whether the hole content behaves like a real occlusion. Iterative Geometry-Aware Cross Guidance Network for Stereo Image Inpainting (IGGNet) does put a cost volume inside an inpainter. It uses it to borrow pixels from the other view for synthetic random masks. That is the opposite regime: a true disocclusion has nothing to borrow. Our proposal differs from IGGNet in using the matcher to verify the output, not to source it.

Research Program

The system has three parts.

The judge. A frozen occlusion-aware stereo network $F$ with a GOAT-style disparity path and a separate occlusion path. We train it on Scene Flow, which has ground-truth occlusion masks. We then fine-tune it on real rectified stereo, meaning image pairs aligned so that matching points lie on the same horizontal row. The real data are Stereo4D and KITTI 2015, with occlusion pseudo-labels from left-right consistency checks, as GOAT itself does. Run with the generated right view $\hat R$ as reference and the real left view $L$ as target, it returns a disparity map $\hat d$ and an occlusion probability map $\hat o$.

The generator. A fast feed-forward stereo inpainter in the style of GRT's image and video models. It takes the depth-warped right view and the disocclusion mask $M$, and is trained with GRT's self-supervision on monocular video. We start from the GRT architecture because it is fast (0.05 s per frame reported), which makes a judge-in-the-loop affordable.

The added objective. Let $d^\ast$ be the intended target disparity and $V$ the non-disoccluded pixels. The loss is

$$\mathcal{L}_{\text{judge}} = \frac{1}{|V|}\sum_{p\in V} |\hat d(p) - d^\ast(p)| \;+\; \beta\,\mathrm{BCE}(\hat o, M).$$

The first term asks that the matcher see the intended depth on visible pixels. It penalizes identity copying, which yields $\hat d \approx 0$. The second term asks that the matcher find no partner inside the hole, and does find partners outside it. Here $\beta$ weights the occlusion term and BCE is per-pixel binary cross-entropy.

The occlusion term is the heart of the proposal. A disparity-only term is nearly blind inside holes, because a GOAT-style matcher deliberately ignores local costs there and fills depth from context whatever the hole's texture. The occlusion term reads exactly the evidence that texture changes. This yields a clean ablation.

Guarding against gaming. A generator optimized against a frozen network may learn to fool it rather than to be correct. We therefore never evaluate with the network used in training. Evaluation uses held-out judges of different design: an AANet-style matcher with adaptive 2D aggregation (AANet: Adaptive Aggregation Network for Efficient Stereo Matching), a DiffuVolume-refined cost-volume matcher (DiffuVolume: Diffusion Model for Volume based Stereo Matching), and a classical left-right consistency check on semi-global matching, which has no learned parameters to exploit. Gains that appear only under the training judge count as failure.

The metric. The same held-out judges define an evaluation quantity. The first part is hole-occlusion agreement: the IoU between the judge's occlusion map and $M$. The second is the spurious-match rate: the fraction of hole pixels where the judge reports a confident match. Before this metric is trusted, it must pass two calibration checks. Real stereo pairs from Stereo4D, with ground-truth right views and matcher-derived masks, should score near the top. Left-view copies should score near the bottom.

Baselines

Naive reference points. The weakest reference simply copies the left view as the right view. It anchors the identity-mapping trap and should score well on PSNR and badly on every judge metric. A second reference is depth warping followed by classical diffusion-style (Navier–Stokes) image inpainting, with background disparity extrapolated across holes as in GOUSM's post-processing (Geometry-based Occlusion-Aware Unsupervised Stereo Matching for Autonomous Driving). This shows what non-learned filling does under the judge. Both references use the same monocular depth estimator and masks as every other method.

Strongest existing method. This is the GRT feed-forward inpainter, retrained by us on the same monocular video data (ImageNet-GRT and Kinetics-GRT style), depth estimator, resolution, and training steps as our model. The only difference is $\mathcal{L}_{\text{judge}}$. GRT reports PSNR 35.52 (image model) and 34.06 (video model) on DAVIS-GRT, against roughly 28–29 for StereoCrafter and StereoDiffusion. It is the strongest reported number in the file for disocclusion inpainting. It is also the closest competitor because it already has true pixel targets inside holes. For M2SVid we could find no reported PSNR or LPIPS values on Stereo4D in the material, only a DAVIS user-study average rank of 1.43 among four methods. We will run M2SVid as released, or re-implemented, as a second strong baseline. Mono2Stereo's dual-condition model with its edge-consistency loss is a third. For Mono2Stereo we found no reported raw SIoU values on Lacuna.

What beating them means. We need a clear improvement in held-out hole-occlusion agreement and spurious-match rate, together with a human-rated reduction in rivalry. PSNR on DAVIS-GRT must stay within about 0.5 dB of retrained GRT, and hole-region LPIPS on Stereo4D must be no worse.

Ablations. These isolate the key ingredient. We train GRT plus the disparity term alone ($\beta = 0$), GRT plus the occlusion term alone, and GRT plus Mono2Stereo's edge-consistency loss in place of ours. All use an identical tuning budget: the same sweep of three loss weights for each.

Why not just…?

Why not just add a disparity-consistency loss from a frozen stereo network? This is the obvious version, and we include it as an ablation. It is unlikely to fix holes. Occlusion-aware matchers such as GOAT fill disparity inside occluded regions from global context rather than local matching evidence. The predicted disparity in a hole therefore barely depends on what was painted there. The $\beta = 0$ ablation tests this directly. If it matches the full model on held-out hole-occlusion agreement, the occlusion term is unnecessary and the proposal reduces to this simpler idea.

Why not just rely on GRT's self-supervision, which already supervises hole pixels? This is the most serious threat. GRT's targets are real pixels, and real pixels behave like real occlusions. It has two weaknesses. Its masks depend on a monocular depth estimator whose edge errors corrupt them, which the GRT paper acknowledges. And pixel losses on uncertain regions reward blurry averages, which can themselves produce weak false matches. The decisive test comes first in our schedule. We measure held-out judge metrics on retrained GRT outputs against real Stereo4D pairs. If GRT is already indistinguishable from real stereo on these metrics, we stop and report that as a finding.

Why not just use SIoU or an edge-consistency loss? SIoU and Mono2Stereo's Sobel-based loss score edge maps and difference maps between the two views. They reward correct boundary shifts, but they say little about whether the interior texture of a hole has a spurious partner along its horizontal line. We replace the judge loss with the edge loss under equal tuning, and report both SIoU and judge metrics for every method. This shows whether the two capture the same errors.

Why not just run a left-right consistency check as the metric? A classical check is included as one of the held-out judges precisely because it cannot be gamed. It gives binary, noisy verdicts in textureless regions, as the KITTI confidence analysis (Analysis of KITTI Data for Stereo Analysis with Stereo Confidence Measures) documents. That is why it is one judge among several rather than the training signal.

Proposed Experiments

The first experiment validates the metric. On Stereo4D and KITTI 2015 we compute held-out judge scores for real right views, left copies, warp-plus-DeepFill-style outputs, GRT, M2SVid, and Mono2Stereo. We then run a human study on Mono2Stereo and Inria 3DMovie clips shown on a stereo display. Participants rate rivalry and depth plausibility in disoccluded regions. We compare the Spearman correlation of our metric with these ratings against SIoU's reported 0.84, restricted to hole regions. The metric needs to add information beyond SIoU, not merely match it.

The second experiment is the main comparison. We compare the retrained GRT baseline, M2SVid, Mono2Stereo, the ablations, and our model. The measures are PSNR on DAVIS-GRT, hole-region PSNR and LPIPS on Stereo4D ground-truth right views, SIoU on Mono2Stereo across its Indoor, Outdoor, Complex and Animation splits, and held-out judge metrics everywhere. We also stratify results by thin-structure holes, such as poles and chair legs, and by repetitive texture. The false-target problem (A Segment-Based Matching Algorithm in Trinocular Vision) predicts that spurious matches concentrate in repetitive texture, and that is where we expect the largest gains.

The third experiment is the gaming check. We report the gap between training-judge and held-out-judge scores, and we require the human study to confirm any held-out gain.

The decisive outcome is simple. The proposal succeeds if the full loss beats both retrained GRT and the disparity-only ablation on held-out judges and in human rivalry ratings, at matched pixel fidelity. It fails, informatively, if GRT already scores like real stereo or if the gains appear only under the training judge.