Extreme Image Inpainting

337 concepts239 papers4 proposals

Image inpainting is trivial when the missing region is small. If a few pixels are scratched out of a brick wall, a model can look at the immediate neighbors and copy the texture inward. This process relies entirely on local continuity.

However, when an occlusion scales up to cover a massive portion of the image—such as removing a large foreground object or repairing a heavily corrupted sensor array—the problem undergoes a fundamental phase transition. It shifts from a low-level texture-matching task to an ill-posed semantic hallucination problem.

As noted by Jiahui Yu et al. in Generative Image Inpainting With Contextual Attention, large missing regions exhibit extreme semantic ambiguity at their centers. The core issue is contextual information scarcity: as the distance from the known boundary pixels increases, the mathematical constraint exerted by those boundaries decays. At the center of a large hole, the model has no local context to guide it, making an infinite number of completions plausible.

Why Local Operators Fail at Scale

Traditional inpainting techniques rely on local operations to fill gaps. For example, partial convolutions, introduced by Fitsum Reda et al. in Image Inpainting for Irregular Holes Using Partial Convolutions, dynamically mask and normalize convolution operations so they only depend on valid pixels. While highly effective for irregular streaks or small gaps, this mechanism struggles when the missing region is vast. The valid features must propagate layer-by-layer from the boundaries into the center. If the hole's diameter exceeds the network's receptive field, the information dilutes before reaching the center, failing to reconstruct complex semantic details.

This breakdown is not unique to convolutional networks. Even modern generative architectures face a steep performance drop-off when the missing region crosses a critical threshold—typically around 50% of the image area, as documented by Manyu Zhu et al. in Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness and Ye Deng et al. in T-former: An Efficient Transformer for Image Inpainting.

When faced with high-occlusion scenarios, attention-based mechanisms in diffusion models can also falter. Yi Liu et al. observe in Erase Diffusion: Empowering Object Removal Through Calibrating Diffusion Pathways that without sufficient surrounding context, attention layers often fail to reconstruct the background, stubbornly regenerating variations of the occluding object instead of erasing it.

Structural Anchoring

To overcome contextual scarcity, modern frameworks decouple the problem into two distinct phases: structural hallucination and texture synthesis. Instead of trying to guess pixels directly, the model first attempts to draw the global geometric "skeleton" of the missing scene.

By predicting low-frequency structural priors (such as edges, contours, or line continuities), the model effectively breaks a massive, unmanageable hole into several smaller, well-bounded sub-regions. Chenjie Cao et al. demonstrate this in ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors, showing that establishing global architectural geometry from sparse anchor points prevents structural collapse.

Similarly, Dongli Jia in A Generate Adversarial Network with Structural Branch Assistance for Image Inpainting utilizes a dedicated structural branch to guide a texture generation branch, ensuring that the final output maintains both global geometric coherence and local textural plausibility.

[Large Masked Image] 
       │
       ▼
[Structural Branch] ───► Hallucinates Global Edges & Contours (Anchors)
       │
       ▼
[Texture Branch]    ───► Fills Sub-regions with High-Frequency Details

Iterative Shrinking and Confidence Loops

Another powerful intuition for solving extreme inpainting is to transform a single, highly difficult step into a sequence of progressively easier steps. Training a network to fill a massive hole all at once often leads to poor convergence because the difficulty curve is too steep, a limitation discussed by Soo Ye Kim et al. in Zoom-to-Inpaint: Image Inpainting with High Frequency Details.

Instead, models can use iterative confidence feedback loops. As designed by Yu Zeng et al. in High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling, the network generates a rough completion but only commits to the pixels near the boundaries where its confidence is high. It then updates the mask, treats these newly generated pixels as valid context, and runs another pass. This iterative shrinking mechanism brings reliable contextual information closer and closer to the center, mitigating the ambiguity of wide-area completion by transforming an extreme hole into a series of shallow ones.

Resolving Ambiguity with Semantic Priors

When an occlusion is so severe that even global geometric cues are missing—such as when an entire face or a specialized object is completely obscured—the problem becomes fundamentally impossible to solve using the image's remaining pixels alone. In these extreme scenarios, the model requires external world knowledge to resolve semantic ambiguity.

Yongsheng Yu et al. address this in Unbiased Multi-modality Guidance for Image Inpainting by incorporating multi-modality guidance, such as semantic segmentation maps, to constrain the generative space.

When coordinate-based neural representations completely lose local appearance cues due to large-scale failures, Canyu Zhang et al. show in SAIR: Learning Semantic-Aware Implicit Representation that injecting text-aligned semantic embeddings (e.g., from CLIP) provides the high-level conceptual guidance necessary to hallucinate anatomically or logically correct features out of thin air. By anchoring the reconstruction process to an explicit semantic concept, the model no longer guesses blindly; it synthesizes content that conforms to a known global context.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_contextual_information_scarcity_d4b5c4c8,
  title = {contextual information scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contextual-information-scarcity/rdr_d4b5c4c8ad834a65bb4e503f2a0f292b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_contextual_information_scarcity_d4b5c4c8_v1,
  title = {contextual information scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/contextual-information-scarcity/rdr_d4b5c4c8ad834a65bb4e503f2a0f292b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake