Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image

Yinan Zhao, Brian L. Price, Scott Cohen, Danna Gurari

TL;DR

Most image inpainting tools "hallucinate" missing pixels based on the surrounding context, giving users little control over the result. This paper introduces an end-to-end deep learning framework that allows users to guide the inpainting process by providing a second "guidance image" to fill the hole, using a novel attention mechanism to seamlessly blend inconsistent textures and lighting.

What Problem Does This Paper Address?

Traditional image inpainting (or hole filling) typically relies on two approaches:

  1. Diffusion/Texture Synthesis: Propagating surrounding pixels into the hole.
  2. Deep Generative Models: Using neural networks to predict what should be there based on learned patterns.

The problem? Users can't choose the content. If you want to replace a passerby in a photo of a city with a specific beach scene from your vacation, standard tools won't help. While "cut-and-paste" methods exist, they often result in jarring edges, mismatched lighting, and perspective errors. The authors tackle the challenge of Guided Image Inpainting: how to pull content from a different image and synthesize it so it looks like it was always part of the original scene.

Methods

The authors propose a two-stage neural architecture:

  1. Localization Network: This module acts as a "Spatial Transformer." It takes the incomplete image and the guidance image, then regresses six affine transformation parameters to automatically crop, rotate, and scale a patch from the guidance image to fit the hole.
  2. Synthesis Network: This is an encoder-decoder CNN with a twist. It features three branches:
  3. Context Branch: Encodes the surrounding area of the original image.
  4. Guidance Branch: Encodes the aligned patch from the guidance image.
  5. Attention Branch: This is the core innovation. It produces a spatial map indicating how well each pixel in the guidance patch fits the original context. It tells the network: "Keep this part of the guidance image, but synthesize something new here to fix the transition."

Training Strategy: Since there is no "ground truth" for replacing one image with another, the authors trained the model on synthetic data. They took an image, cropped a patch, corrupted its color/lighting, and tasked the network with restoring it to the original.

Loss Functions: The model uses a combination of Perceptual Loss (comparing high-level features via a VGG network) and Adversarial Loss (using a GAN with both global and local discriminators to ensure the results look "real").

Datasets

  • ADE20K: Used for the bulk of training and quantitative evaluation. The authors generated over 1 million synthetic training pairs from this dataset.
  • MSCOCO: Used during the evaluation phase to provide a diverse pool of real-world guidance images for user studies.

Key Insights and Ideas

  • The Inconsistency Challenge: The hardest part of guided inpainting isn't the middle of the hole; it's the boundary. The "Attention Branch" is a clever way to handle regions where the guidance image and the original image simply don't match, allowing the network to prioritize context over guidance when necessary.
  • Synthetic to Real Generalization: A major "aha" moment is that a network trained only to "fix" corrupted versions of the same image can generalize effectively to blending content from entirely different images.
  • Dual Critics: By using both a global discriminator (looking at the whole image) and a local discriminator (zooming in on the patch), the model ensures the new content fits the overall scene logic while maintaining sharp, realistic textures at the edges.

Key Results

The authors compared their method against seven baselines, including classic methods like Poisson Blending and modern deep learning models like Context Encoders.

  • Quantitative: In image restoration tasks, the model achieved a PSNR of 20.68dB, significantly outperforming the best baseline (16.55dB).
  • Absolute Realism: In a "Turing test" for images on Amazon Mechanical Turk, users mistook the model's results for real photos 33-36% of the time. While this shows there is still room for improvement, it was nearly double the performance of several deep learning baselines.
  • Relative Realism: When shown side-by-side comparisons, human participants preferred the authors' results over the baselines in at least 66% of test cases.

Limitations and Future Work

  • Complex Structures: The model struggles to synthesize complex, high-level structures (like cars or people) from scratch if the guidance image is a poor match. If the guidance patch is missing a wheel, the network might not be able to "invent" a realistic one to bridge the gap.
  • Localization Generalization: The localization network (finding the right patch) did not generalize to natural images as robustly as the synthesis network. The authors suggest that the distribution of features used for alignment in synthetic training differs too much from real-world image pairs.
  • Future Direction: Improving the localization network to handle more complex deformations (beyond affine transforms) and training on more diverse semantic pairings could further enhance realism.

Summary updated

Abstract

Deep generative models have shown success in automatically synthesizing missing image regions using surrounding context. However, users cannot directly decide what content to synthesize with such approaches.We propose an end-to-end network for image inpainting that uses a different image to guide the synthesis of new content to fill the hole. A key challenge addressed by our approach is synthesizing new content in regions where the guidance image and the context of the original image are inconsistent. We conduct four studies that demonstrate our method yields more realistic image inpainting results over seven baselines.
BibTeX
@inproceedings{zhao2019guided,
  author = {Yinan Zhao and Brian L. Price and Scott Cohen and Danna Gurari},
  title = {{Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=tOXFQl583tV},
}

Loading related content...

Figure