From Missing Pieces to Masterpieces: Image Completion With Context-Adaptive Diffusion

Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Michael Felsberg, Dacheng Tao, Xuelong Li

Abstract

Image completion, or inpainting, aims to restore missing regions of an image with content that is both visually plausible and structurally consistent with the surrounding context. While modern diffusion models excel at generating high-quality textures, they often struggle with precise contextual alignment, leading to visible artifacts or logical inconsistencies at mask boundaries. We introduce ConFill, a novel framework that addresses these challenges through Context-Adaptive Discrepancy (CAD) and Dynamic Sampling. By leveraging optimal transport principles and local feature awareness, ConFill ensures that generated patches seamlessly integrate with the original image's structural "DNA." Our results demonstrate state-of-the-art performance across multiple benchmarks, significantly outperforming existing diffusion-based methods in both perceptual metrics and human preference.

Introduction

The primary challenge in image completion is ensuring that "hallucinated" content matches the specific, localized features of the target image—such as unique textures, lighting, and structural flow. Existing methods often fail due to two main issues:

  1. Metric Mismatch: Reliance on Euclidean distance for patch fitting often ignores complex textures, resulting in low-pixel-error but perceptually jarring results.
  2. Global vs. Local Focus: Standard models prioritize global data distributions over the specific local context of a single image, leading to a lack of "flow" between original and restored regions.

ConFill overcomes these hurdles by shifting from static replacement to an adaptive denoising process that prioritizes contextual coherence.

Methodology

The core of ConFill lies in its ability to adapt the diffusion process to the local environment of the mask.

Context-Adaptive Discrepancy (CAD)

Instead of simple distance metrics, ConFill utilizes CAD, which is grounded in the Brenier potential from Optimal Transport theory. CAD minimizes the discrepancy between latent distributions across timesteps while incorporating a scaling function sensitive to local features like edges and color gradients. This ensures the model matches the underlying structural characteristics of the surrounding pixels rather than just filling space with generic content.

CAD minimizes discrepancy more effectively than standard metrics across denoising steps.

Dynamic Sampling and Gradual Approximation

To handle varying levels of structural complexity, ConFill employs Dynamic Sampling. This mechanism adaptively increases the sampling rate in high-entropy regions (e.g., intricate patterns) while maintaining efficiency in simpler areas (e.g., flat backgrounds). Furthermore, a step-by-step optimization process refines the posterior distribution, reducing the approximation gap inherent in mapping noisy latent spaces to clean images.

Experiments and Results

ConFill was rigorously evaluated on three major benchmarks: CelebA-HQ, Places2, and ImageNet-1K. Testing involved diverse mask configurations, including "Narrow," "Wide," "Half," and "Expand" (outpainting).

Key Findings

  • Perceptual Quality: On ImageNet, ConFill achieved a 10.9% improvement in LPIPS scores over RePaint, indicating superior perceptual similarity to ground truth.
  • Human Preference: User studies revealed a strong preference for ConFill, with participants choosing it 21% more often than CoPaint on Places2 and 29% more often than RePaint on ImageNet.
  • Efficiency: The model demonstrates rapid convergence, achieving a 62-point FID improvement on Places2 within just 50k iterations.
  • Qualitative Excellence: Visual results demonstrate that ConFill maintains sharp edges and consistent lighting even in challenging outpainting tasks where other models produce blurriness or color mismatches.
Qualitative comparison showing ConFill's superior performance across various mask types and tasks.

Conclusion

ConFill represents a significant step forward in image completion by prioritizing local contextual alignment through adaptive diffusion. By moving beyond simple pixel-wise metrics to feature-aware discrepancy measures, it produces results that are not just plausible, but truly seamless. Future work will explore extending this context-adaptive approach to video inpainting to address temporal coherence.

Summary updated

BibTeX
@article{shamsolmoali2025from,
  author = {Pourya Shamsolmoali and Masoumeh Zareapoor and Huiyu Zhou and Michael Felsberg and Dacheng Tao and Xuelong Li},
  title = {{From Missing Pieces to Masterpieces: Image Completion With Context-Adaptive Diffusion}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3558092},
}
Figure