Generative Hallucination for Disoccluded Region Synthesis

1,194 concepts862 papers44 proposals

When we change our perspective on a scene—whether by moving a camera or watching an object rotate—our eyes receive new visual information. Surfaces that were previously hidden behind foreground objects or obscured by self-occlusion suddenly become visible. This phenomenon is known as disocclusion.

In computer vision tasks like novel view synthesis, 3D reconstruction, and pose-guided human image generation, handling disocclusions is a fundamental hurdle. Pure geometry can tell us where a hole will appear when a viewpoint shifts, but it cannot tell us what is inside that hole. To create realistic images, models must learn to hallucinate missing textures and structures in unobserved regions while maintaining strict consistency with the visible parts of the scene.

The Geometry of Disocclusion

To understand why this is difficult, consider traditional spatial transformation techniques. Methods like appearance flow (e.g., View Synthesis by Appearance Flow) operate by predicting how pixels from a source image should be relocated to form a target image. This works beautifully for surfaces that remain visible across both views.

However, when a camera shift or a change in human pose reveals a previously occluded region, the source image contains exactly zero pixels of that surface. Because pure geometric warping can only rearrange existing data, it fails catastrophically at disocclusion boundaries, resulting in severe texture stretching or empty "holes" (Srinivasan et al.; Asthana et al.).

Source View:       [ Foreground Object ] -------> Blocks Background
                      \               /
Target View:           \             /  -------> Reveals "Hole" (Disocclusion)
                        v           v
                 [ Missing Background Texture ]

Even modern, high-fidelity 3D reconstruction frameworks suffer from this limitation. Optimization-based representations like Neural Radiance Fields (NeRFs) or pixel-aligned Gaussian Splatting excel at synthesizing views within their training trajectories, but they rely heavily on multi-view consistency. When asked to render a permanently occluded region, they lack the generative capacity to invent plausible textures, often producing blurry artifacts or "floaters" instead (OmniLocalRF; Zhang et al.).

Decomposing Warping and Inpainting

To bypass this geometric bottleneck, modern architectures decouple the problem into two distinct operations: spatial transformation for visible content and generative inpainting for disoccluded content.

Instead of forcing a single network to perform both tasks end-to-end, frameworks like the one proposed by Balakrishnan et al. explicitly segment the pipeline. The model first warps the visible source pixels to their estimated target locations. This leaves well-defined empty regions (holes) where disocclusions occurred. A secondary generative network—often trained with adversarial or reconstruction losses—then focuses entirely on filling these holes based on contextual clues from the surrounding environment.

In articulated contexts like human pose transfer, Siarohin et al. implemented deformable skip connections to warp feature maps according to body movements. This architecture allows the network to preserve fine-grained identity details in visible regions via direct feature routing, while giving the decoder the flexibility to synthesize entirely new feature patches when hidden body parts or background elements are exposed.

Anchoring Hallucinations with Structural Priors

Unconstrained generative hallucination can easily lead to visual inconsistencies. For example, if a model hallucinates the back of a person's shirt, it needs to know whether to continue a pattern from the front or leave it blank. To prevent arbitrary or unrealistic generations, researchers anchor the hallucination process using structural and statistical priors.

3D Scaffolding

One powerful approach is to use explicit 3D geometry as a guide for texture synthesis. Rematas et al. demonstrated that aligning class-specific 3D CAD models to a single 2D image provides a geometric scaffold. This scaffold dictates the boundaries of hidden object parts, turning an unconstrained image-generation problem into a bounded texture-completion task.

Similarly, in human reposing, Sanyal et al. leveraged parametric 3D body models (SMPL) to enforce shape and appearance consistency. By mapping the source image onto a 3D surface, the model can reason about self-occlusions explicitly, reducing the ambiguity of predicting how textures should wrap around unobserved surfaces.

Statistical Feature Consistency

When explicit 3D models are unavailable, networks can rely on the internal feature correlations of deep architectures. For high-fidelity facial texture inference, Wei et al. utilized the style-distribution properties of deep layers to synthesize mesoscopic details (such as pores and wrinkles) across occluded regions. By forcing the hallucinated textures to share statistical feature distributions with the visible regions, the synthesized surfaces match the lighting, resolution, and style of the original capture.

Advanced Representations: From Coordinates to Diffusion

Directly generating RGB pixels in disoccluded regions frequently results in blurry or low-frequency patches because the network tries to average all possible valid answers to minimize regression losses. To maintain sharpness, modern approaches have shifted toward more sophisticated target representations.

Coordinate Inpainting

Rather than predicting color values directly, Grigorev et al. designed a system that inpaints source image coordinates (UV maps). When a disocclusion occurs, the network predicts which visible part of the source image is texturally closest to the hidden region. The system then samples pixels directly from those coordinates. This ensures that even hallucinated regions retain the sharp, high-frequency details of the original photograph.

Generative Diffusion Priors

The rise of large-scale generative models has provided a massive leap in disocclusion quality. Instead of learning scene priors from scratch on small datasets, recent view-synthesis methods inject powerful diffusion models directly into the geometric pipeline.

For instance, Pu et al. combined Stable Diffusion with Multiplane Images (MPIs) to fill out the edges and hidden layers of a scene during wide-baseline camera movements. In the temporal domain, Bhat repurposed video diffusion models to instantly inpaint disocclusion holes when converting monocular video into stereoscopic 3D. By conditioning the diffusion process on the original view, these models leverage billions of learned real-world patterns to generate structurally sound, temporally consistent, and visually stunning completions of the unseen world.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_disocclusion_artifacts_36cb2ab8,
  title = {disocclusion artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disocclusion-artifacts/rdr_36cb2ab87c674dc29a2410ae61607b17},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_disocclusion_artifacts_36cb2ab8_v1,
  title = {disocclusion artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disocclusion-artifacts/rdr_36cb2ab87c674dc29a2410ae61607b17/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake