Occlusion Aware Visibility Mapping for Image and Video Synthesis

46 concepts38 papers

When synthesizing novel views of a 3D scene or interpolating intermediate frames in a video, the most efficient strategy is to reuse existing pixels. By computing spatial or temporal correspondences—such as optical flow, appearance flow, or coordinate offsets—we can warp pixels from reference images directly to their new destinations.

However, this elegant shortcut completely breaks down at occlusion boundaries. When an object moves or a camera shifts its perspective, certain regions of the scene disappear while others are revealed (disocclusion). Traditional warping algorithms assume every target pixel can find a match in the source frame. For a newly revealed background pixel, no such match exists. Forcing a correspondence where none exists results in classic visual artifacts: ghosting, smearing, and unnatural stretching.

To solve this, a powerful paradigm has emerged: occlusion-aware visibility mapping. By explicitly modeling which regions are visible, hidden, or unreliable, modern synthesis networks can dynamically alter their behavior on a per-pixel level, switching seamlessly between copying valid pixels and generating new ones.


Detecting the Blind Spots via Consistency and Geometry

How can a system know when a pixel's mapping is a lie? The most direct approach relies on geometric or flow consistency checks.

Consider bidirectional optical flow. If we track a pixel forward from Frame A to Frame B, and then track it backward from Frame B to Frame A, the paths should be perfectly symmetrical. If the forward and backward vectors disagree, the pixel has likely been occluded or altered. The authors of Hand-held Video Deblurring via Efficient Fourier Aggregation leverage this exact bidirectional consistency check to identify unreliable pixel alignments, treating inconsistent regions as occlusions to prevent ghosting during frame fusion.

This principle translates naturally into deep learning architectures. For instance, Li et al. (Deep Sketch-Guided Cartoon Video Inbetweening) calculate flow consistency to isolate occluded regions, generating a mask that guides selective pixel fusion. In multi-dimensional interpolation frameworks, Bemana et al. (X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation) down-weight regions where flow predictions from multiple reference images disagree, ensuring that conflicting spatial data does not corrupt the final composition.


Soft Visibility Maps and Multi-View Confidence Blending

Instead of making binary true/false decisions about pixel validity, modern architectures learn continuous visibility maps or confidence maps. These maps act as soft masks, allowing the network to gracefully weight and blend information from multiple reference perspectives based on how "reliable" each source is.

[Source Frame A] ---> Warping ---> [Warped A] ---\
                                                   ---> (Weighted Blending) ---> [Synthesized Frame]
[Source Frame B] ---> Warping ---> [Warped B] ---/           ^
                                                             |
[Source Frames]  ---> Visibility Net ---> [Soft Mask] -------/

In video frame interpolation, Jiang et al. (Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation) pioneered the use of learned visibility maps to weight the contributions of past and future frames. If an object is about to be occluded in a future frame, the visibility map down-weights that frame's contribution at that specific timestamp, forcing the system to rely entirely on the past frame where the region is still visible.

This mechanism is equally vital when synthesizing novel viewpoints from sparse camera arrays. When cameras are spaced far apart, an object might completely block a background feature from Camera A's perspective while Camera B sees it perfectly. Zhou et al. (View Synthesis by Appearance Flow) address this by predicting a per-pixel confidence mask to intelligently select the most unobstructed source pixels.

Subsequent work has refined this intuition; Liu et al. (Learning Dynamic Interpolation for Extremely Sparse Light Fields with Wide Baselines) and Guo et al. (Content-aware Warping for View Synthesis) demonstrate that learned confidence maps drastically reduce boundary artifacts by prioritizing the specific camera view that holds the clearest, most direct line of sight to a target spatial coordinate.


Gating Warping vs. Generative Hallucination

What happens when a region is completely occluded across all available reference frames? No amount of clever blending can recover a pixel that was never captured by any camera. In these scenarios, the visibility map serves a higher conceptual purpose: it acts as a gating mechanism that tells the network when to stop copying pixels and start hallucinating them.

                  +-------------------------+
                  |  Is Region Visible?     |
                  +------------+------------+
                               |
              +----------------+----------------+
              |                                 |
           [ YES ]                           [ NO ]
              |                                 |
              v                                 v
   +--------------------+             +--------------------+
   | Warp Source Pixels |             | Activate Generative|
   |   (High Fidelity)  |             |   Context Prior    |
   +--------------------+             +--------------------+

Huang et al. (Dense Intrinsic Appearance Flow for Human Pose Transfer) explicitly implement this gating strategy by predicting a visibility map alongside spatial flow. This map determines whether a target body part can be populated via direct texture warping or if it must be generated from scratch due to a change in body pose. In a similar vein, Ding et al. (CDFI: Compression-Driven Network Design for Frame Interpolation) utilize a learned occlusion mask to drive a path selection mechanism, dynamically balancing a warping module for visible regions and a synthesis module for hidden ones.

For reference-guided image inpainting, Zhou (TransFill: Reference-Guided Image Inpainting by Merging Multiple Color and Spatial Transformations) demonstrates that a learned per-pixel confidence map allows a system to seamlessly transition from copying high-fidelity reference data to generating single-image inpainted content when spatial alignment fails. Even in highly structured domains like face reconstruction, Zhu et al. (Deep Cascaded Bi-Network for Face Hallucination) use a gated fusion mechanism to balance visible textures with learned structural priors, preventing the network from generating false features over physical obstructions like eyewear.


Extending Visibility Mapping to Latent Spaces and Diffusion Models

As the field has transitioned from direct pixel manipulation to generative diffusion models, the core intuition of occlusion-aware visibility mapping has remained remarkably robust. Instead of masking raw pixels, modern frameworks apply visibility masks to latent features and attention tokens to maintain temporal consistency.

In zero-shot video translation, Bao et al. (LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation) introduce an occlusion-aware binary mask to identify where motion estimation breaks down. This mask dictates where the system can safely propagate warped historical diffusion latents to maintain temporal consistency, and where it must allow the diffusion model to generate entirely new content. Taking this a step further, Zhu et al. (Zero-Shot Video Translation via Token Warping) apply occlusion masks directly to the query, key, and value tokens within a transformer architecture, preventing corrupt warped tokens from introducing artifacts into newly revealed regions.

Finally, Chen et al. (UltraFusion: Ultra High Dynamic Imaging using Exposure Fusion) demonstrate how this concept empowers high dynamic range (HDR) imaging. By using optical flow consistency checks to flag occluded or misaligned regions across different exposures, their system generates a soft mask that instructs a generative diffusion model exactly where to prioritize synthesis over direct pixel transfer, eliminating the ghosting artifacts that plague traditional exposure blending.

Key Foundational Work

Optical flow based super-resolution: A probabilistic approach (2007) — contributed 1 concept elements to this direction.
Discontinuity preserving disparity estimation with occlusion handling (2014) — contributed 1 concept elements to this direction.
Sparse Coding Approach for Multi-Frame Image Super Resolution (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_occlusion_aware_visibility_mapping_5ebf69c9,
  title = {occlusion-aware visibility mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/occlusion-aware-visibility-mapping/rdr_5ebf69c9171a4207864b75e9bc831232},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_occlusion_aware_visibility_mapping_5ebf69c9_v1,
  title = {occlusion-aware visibility mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/occlusion-aware-visibility-mapping/rdr_5ebf69c9171a4207864b75e9bc831232/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake