Abstract
Deep generative models have shown success in automatically synthesizing missing image regions using surrounding context. However, users cannot directly decide what content to synthesize with such approaches. We propose an end-to-end network for image inpainting that uses a different image to guide the synthesis of new content to fill the hole. A key challenge addressed by our approach is synthesizing new content in regions where the guidance image and the context of the original image are inconsistent. We conduct four studies that demonstrate our results yield more realistic image inpainting results over seven baselines.
Introduction: The Challenge of User-Controlled Inpainting
Traditional image inpainting (or hole filling) typically relies on two primary approaches: diffusion-based texture synthesis, which propagates surrounding pixels into the hole, and deep generative models, which use neural networks to predict missing content based on learned patterns. While these methods are effective for background completion, they offer users little control over the semantic content of the result. For instance, replacing a specific object in a photograph with an element from a different scene remains a difficult task. Standard "cut-and-paste" methods often result in jarring edges, mismatched lighting, and perspective errors.
This paper introduces a framework for Guided Image Inpainting, enabling users to pull content from a separate "guidance image" to fill a hole in a target image. The goal is to synthesize the new content so that it appears as a seamless, natural part of the original scene.
Methodology: A Two-Stage Synthesis Architecture
The authors propose an end-to-end neural architecture consisting of two main components: a Localization Network and a Synthesis Network.
1. Localization Network
This module functions as a "Spatial Transformer." It receives both the incomplete target image and the guidance image, regressing six affine transformation parameters. These parameters allow the network to automatically crop, rotate, and scale a patch from the guidance image to fit the target hole's geometry and perspective.
2. Synthesis Network
The synthesis network is an encoder-decoder CNN featuring a unique three-branch structure:
- Context Branch: Encodes the surrounding area of the original image to maintain global consistency.
- Guidance Branch: Encodes the aligned patch from the guidance image.
- Attention Branch: This is the core innovation. It produces a spatial map indicating how well each pixel in the guidance patch fits the original context. This allows the network to selectively keep parts of the guidance image while synthesizing new transitions where the source and target are inconsistent.
Training Strategy and Loss Functions
Because no "ground truth" exists for replacing content between two different images, the model is trained on synthetic data. The researchers took single images, cropped patches, corrupted their lighting and color, and tasked the network with restoring them using the original uncorrupted patch as guidance.
To ensure high-quality results, the model employs a combination of Perceptual Loss (using a VGG network to compare high-level features) and Adversarial Loss. The adversarial component uses "Dual Critics"—a global discriminator to ensure overall scene logic and a local discriminator to maintain sharp, realistic textures at the boundaries of the inpainted region.
Results and Evaluation
The proposed method was evaluated against seven baselines, including classic Poisson Blending and modern deep learning models like Context Encoders, using the ADE20K and MSCOCO datasets.
Quantitative Performance
In image restoration tasks, the model achieved a PSNR of 20.68dB, significantly outperforming the best baseline (16.55dB). This demonstrates the effectiveness of the attention mechanism in reconstructing corrupted image data.
Human Perception Studies
In "Turing tests" conducted via Amazon Mechanical Turk, participants mistook the model's results for real photos 33-36% of the time—nearly double the performance of several deep learning baselines. Furthermore, in side-by-side comparisons, human participants preferred the authors' results over the baselines in at least 66% of test cases.
Key Insights and Limitations
A major finding of this research is that a network trained only to "fix" corrupted versions of the same image can generalize effectively to blending content from entirely different images. The "Attention Branch" proves vital for handling the "Inconsistency Challenge," where the guidance image and the original context do not match perfectly at the boundaries.
However, limitations remain. The model struggles to synthesize complex structures (like wheels on a car) if they are missing or poorly represented in the guidance image. Additionally, the localization network's ability to find the correct patch in natural images is less robust than the synthesis network's ability to blend it. Future work will explore more complex deformations beyond affine transforms and training on more diverse semantic pairings to further enhance realism.
Summary updated