When a computer vision model encounters missing data—whether a large chunk of an image removed during inpainting, or sparse gaps in a depth map—it faces a fundamental challenge: the blurry center problem.
If a model attempts to predict the entire missing region in a single forward pass, it lacks the local context required to reconstruct complex internal structures. Standard convolutional layers apply fixed weights across valid and invalid pixels alike. When a convolutional kernel sits over a large hole, it multiplies its weights by uninformative placeholder values (such as zeros). As data flows deeper into the network, these corrupted features propagate, causing the center of large gaps to become structurally incoherent and blurry.
To solve this, a powerful paradigm has emerged: Progressive Boundary Refinement. Instead of filling a hole all at once, models treat hole filling as an iterative puzzle, systematically shrinking the missing region from the outer boundaries inward. By prioritizing areas with the highest informational certainty—the pixels directly adjacent to known data—the model establishes stable structural anchors before attempting to synthesize the more difficult central regions.
Layer-by-Layer Mask Propagation
The most granular implementation of this idea occurs at the operator level within a neural network's layers. Rather than treating the input image as a static grid, the network dynamically tracks which pixels are valid and which are missing using a parallel binary observation mask.
In Image Inpainting for Irregular Holes Using Partial Convolutions, Guilin Liu addresses the corruption of features by conditioning the convolution operation strictly on valid pixels. The partial convolution re-normalizes the output based on the fraction of valid inputs within the current receptive field:
$$x' = \begin{cases} \mathbf{W}^T (\mathbf{X} \odot \mathbf{M}) \frac{1}{\sum \mathbf{M}} + b & \text{if } \sum \mathbf{M} > 0 \\ 0 & \text{otherwise} \end{cases}$$
where $\mathbf{X}$ represents the feature values, $\mathbf{M}$ is the corresponding binary mask, and $\mathbf{W}$ represents the filter weights. Crucially, the author introduces an automatic mask-updating rule for the next layer:
$$m' = \begin{cases} 1 & \text{if } \sum \mathbf{M} > 0 \\ 0 & \text{otherwise} \end{cases}$$
This simple rule ensures that if a convolutional filter overlaps with even one valid pixel in its current window, the corresponding location in the output mask is marked as valid ($m'=1$).
Iteration 0 (Large Hole) Iteration 1 (Shrinking) Iteration 2 (Filled)
[ V ][ V ][ V ][ V ] [ V ][ V ][ V ][ V ] [ V ][ V ][ V ][ V ]
[ V ][ ? ][ ? ][ V ] ---> [ V ][ V ][ V ][ V ] ---> [ V ][ V ][ V ][ V ]
[ V ][ ? ][ ? ][ V ] [ V ][ V ][ V ][ V ] [ V ][ V ][ V ][ V ]
[ V ][ V ][ V ][ V ] [ V ][ V ][ V ][ V ] [ V ][ V ][ V ][ V ]
(where 'V' is valid context, and '?' is missing data)
As features move deeper into the network, this mechanism creates a progressive frontier of knowledge. The boundaries of the missing region shrink inward layer by layer, allowing information to propagate seamlessly across large contiguous gaps. A similar approach is utilized by Jonas Uhrig et al. in Sparsity Invariant CNNs to handle sparse depth data, and by Tingxuan Huang et al. in Efficient Indoor Depth Completion Network Using Mask-adaptive Gated Convolution, demonstrating that tracking mask validity layer-by-layer prevents sparse voids from diluting valid geometric features.
Macro-Level Boundary Peeling
While layer-by-layer mask updates excel at capturing local textures, complex global structures—such as long continuous edges or semantic object boundaries—require a macro-level strategy. Instead of confining the progressive refinement to individual layers of a single forward pass, macro-level approaches implement feedback loops across entire network stages or recurrent steps.
To scale this intuition, Seoung Wug Oh et al. developed Onion-Peel Networks for Deep Video Completion. The network explicitly identifies the immediate boundary pixels of a hole and completes only that outer "peel" in a given step. The newly synthesized pixels are then immediately adopted as valid local context for the subsequent iteration. This recursive approach ensures smooth transitions from known boundaries toward the center, avoiding the abrupt structural breaks common in one-shot predictions.
Similarly, Jingyuan Li et al. proposed Recurrent Feature Reasoning for Image Inpainting, which executes this boundary-to-center reasoning directly within the convolutional feature space. By recurrently computing the feature maps and updating the mask over multiple iterations, the network uses previously generated peripheral features as structural constraints to anchor predictions for the more ambiguous central areas.
This multi-stage reduction of error accumulation is further validated by Bo Du in Progressive Reconstruction of Visual Structure for Image Inpainting, where the visual structures (like edges) and features are co-developed as the mask shrinks across distinct network stages. Early statistical foundations for this boundary-to-center propagation can even be seen in non-deep learning methods, such as the Gaussian Mixture Model framework proposed by Yiqing Wang in E-PLE: an Algorithm for Image Inpainting.
From Binary Masks to Soft Confidence and Generative Loops
A limitation of hard binary masks is their assumption that any predicted pixel is perfectly accurate. If a model makes an error on an outer layer, a strict binary update treats that erroneous pixel as absolute truth for the next layer, compounding mistakes as the frontier moves inward. Modern extensions of progressive refinement mitigate this by transitioning from binary masks to continuous confidence maps and generative feedback loops.
In High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling, Yu Zeng et al. design a network that outputs both an image prediction and a continuous confidence map. Instead of keeping everything synthesized during a pass, the system retains only high-confidence pixels. The remaining low-confidence regions are carved back out, forming a new, modified hole for the next iteration. This allows the model to self-correct and progressively fill large gaps based on the reliability of its own synthesis.
[ Forward Pass ] ---> Predicts Content + Confidence Map
^ |
| v
[ Mask Update ] <--- Retain Only High-Confidence Pixels
In complex generative tasks, such as reconstructing occluded objects, this concept evolves into a dialogue between generation and segmentation. In Completing Visual Objects via Bridging Generation and Segmentation, Xiang Li et al. introduce MaskComp, which alternates between generating candidate object completions and segmenting them to extract updated shape priors. By treating mask refinement as an iterative denoising process, the model progressively aligns the generated boundaries with realistic object shapes.
The authors extend a similar unsupervised mechanism in PaintSeg: Painting Pixels for Training-free Segmentation, alternating between inpainting foreground regions and outpainting background regions. By comparing the generated pixels to the original image, the system identifies structural discrepancies, signaling exactly where the mask boundaries must be adjusted to perfectly fit the underlying object.
By breaking down a massive, underdetermined synthesis problem into a sequence of localized, boundary-constrained steps, progressive mask refinement transforms hole filling from a blind guess into a structured, verifiable reconstruction process.
Key Foundational Work
E-PLE: an Algorithm for Image Inpainting (2013) — contributed 1 concept elements to this direction.
Quality-Driven Face Occlusion Detection and Recovery (2007) — contributed 1 concept elements to this direction.