Dense correspondence tasks—whether estimating optical flow, stereo depth, 3D scene flow, or semantic alignment—fundamentally seek to map pixels from a source image to a target image. Traditional formulations operate under an implicit assumption: for every pixel in the source view, a corresponding partner exists in the target view.
However, the physical world routinely violates this assumption through occlusions. When an object moves or the camera shifts perspective, surfaces disappear behind foreground occluders or exit the field of view entirely. As Tola et al. observe, these occluded regions contain pixels with no valid match, presenting a fundamental mathematical challenge to global matching frameworks.
The Cost of Forced Matching
When a dense matching algorithm encounters an unmatchable pixel, it does not gracefully stop. Instead, standard optimization objectives force the model to find a match anyway, selecting the "least bad" option from the target image. This blind forcing introduces severe noise and structural corruption into the resulting estimation:
- Texture Bleeding: Without explicit occlusion reasoning, a model attempts to map background textures onto the foreground occluding object. This creates severe visual artifacts where textures appear to "bleed" across geometric boundaries (Pan et al.).
- Warping Artifacts and Distorted Fields: Forcing matches across significant viewpoint shifts or non-rigid deformations causes severe geometric distortions and unnatural warping artifacts in the estimated flow fields (Lee et al.; Deng et al.).
- Semantic Misalignment: In semantic correspondence, where models match different instances of the same category (e.g., two different cars), viewpoint changes frequently hide key object parts. Forcing a model to find matches for these missing regions introduces noise and reduces overall accuracy (Jeon et al.; Danelljan et al.).
[Source Image] ---> Contains regions that disappear in Target Image
|
v
[Standard Cost Volume] ---> Forces a match for EVERY pixel
|
v
[Resulting Flow/Depth] ---> Texture Bleeding & Warping Artifacts
From Post-Processing to Active Filtering
Historically, dense matching algorithms treated occlusions as an afterthought. As Revaud et al. note, many classical frameworks relied on post-processing or variational smoothing to interpolate missing data after the matching phase, using reciprocal verification (left-right consistency checks) merely to flag errors rather than prevent them from corrupting the optimization.
The shift toward active filtering began by treating occlusions as explicit geometric states within the generative model. Early foundational work by Black et al. introduced Bayesian frameworks that modified likelihood functions to actively exclude pixels predicted to be occluded or disoccluded. Similarly, Huguet et al. integrated explicit visibility functions directly into a unified variational framework for scene flow, preventing hidden regions from introducing errors into global energy minimization.
The Core Mechanism: Cost Volume Masking
Modern deep learning architectures operationalize this insight through cost volume masking. A cost volume is a multi-dimensional tensor that stores the matching affinity or "cost" between every pixel in the source image and its potential candidates in the target image.
If an unmatchable pixel is allowed to contribute to this volume, its noisy affinity scores propagate through subsequent convolutional layers, derailing the final flow or depth regression. The core breakthrough of recent architectures is to dynamically identify these regions and neutralize them before they influence the final estimation.
[Source Features] x [Target Features]
|
v
[Raw Cost Volume] <--- Poisoned by unmatchable regions
|
v <--- Apply Predicted Occlusion Mask (Zero out invalid costs)
[Masked Cost Volume]
|
v
[Clean Flow/Depth Regression]
This is achieved through a multi-step pipeline:
- Predicting Matchability: Networks incorporate dedicated decoders or modules to estimate a dense matchability or visibility map. For instance, Zhou et al. utilize a decoder that predicts the explicit probability of a valid correspondence for every pixel, distinguishing reliable regions from background clutter.
- Zeroing Out Invalid Costs: Once an occlusion map is predicted, it acts as a filter. In the PWOC-3D architecture, matching costs for pixels identified as occluded are explicitly zeroed out within the cost volume. This filtering mechanism removes the noise generated by occluded regions during the matching process.
- Coarse-to-Fine Propagation: To handle complex multi-scale structures, Ouyang et al. demonstrate that an occlusion map predicted at a coarse resolution can be used to mask the cost volume at finer resolutions. This prevents the fine-grained correspondence search from being influenced by points that have disappeared from view.
- Dynamic Reweighting as Self-Attention: Rather than a binary mask, Kong et al. treat the predicted occlusion map as a form of self-attention. By dynamically reweighting the cost volume, the model masks out unreliable matching scores in hidden regions and focuses exclusively on visible pixels, yielding significantly sharper motion boundaries.
Why It Works: Isolating the Optimization
By masking the cost volume, the network fundamentally changes how it handles missing data. Instead of trying to optimize an impossible match, the model isolates the matching phase to visible regions (Schuster et al.).
When the unmatchable regions are zeroed out, they no longer contribute gradients or energy to the matching objective. This allows the network's spatial smoothness priors, context networks, or structural constraints (such as the partial matching mechanisms in functional maps explored by Deng et al.) to cleanly interpolate the motion or depth of occluded zones based entirely on surrounding visible context. By treating visibility as an architectural constraint rather than a post-hoc correction, these frameworks eliminate the root cause of warping and bleeding artifacts.
Key Foundational Work
Probabilistic Detection and Tracking of Motion Discontinuities (1999) — contributed 1 concept elements to this direction.
A Variational Method for Scene Flow Estimation from Stereo Sequences (2007) — contributed 1 concept elements to this direction.