Deep Image Blending

Lingzhi Zhang, Tarmily Wen, Jianbo ShiWACV 2020

TL;DR

This paper introduces a novel two-stage deep learning approach for seamless and stylistically consistent image blending. It addresses the limitations of traditional Poisson blending by incorporating deep feature-based content and style losses, enabling more natural integration of objects into diverse target images, including paintings.

What Problem Does This Paper Address?

Image blending, the task of seamlessly inserting an object from a source image into a target image, is a fundamental operation in image composition. Traditional methods like Poisson blending [23] excel at creating smooth boundaries by enforcing gradient domain consistency. However, they struggle to adapt the object's texture and style to match the target background, often leading to visually jarring results. Furthermore, these methods can sometimes cause the target image's colors to "bleed" into the source object, diminishing its original content. Existing deep learning approaches often require paired training data, limiting their generalization capabilities. This paper aims to bridge this gap by proposing a method that achieves both seamless blending and style consistency without requiring extensive training data, generalizing to various image types.

Methods

The proposed method employs a two-stage deep learning pipeline:

  • Stage 1: Seamless Blending: This stage focuses on creating a seamless boundary and initial integration. It initializes a blending image with random noise and iteratively refines it using a combination of losses:
  • Poisson Gradient Loss: A differentiable approximation of the Poisson equation, enforcing gradient domain consistency between the blended image and the source object, while also considering the target image's boundary.
  • Content Loss: Based on deep features (from a VGG network), this loss ensures semantic similarity between the blending region and the source object.
  • Style Loss: Also based on deep features, this loss encourages the blending region to adopt the style of the target image.
  • Regularization Losses: Histogram loss and Total Variation (TV) loss are used to stabilize style transfer and promote spatial smoothness. The optimization in this stage is performed using the L-BFGS solver.

  • Stage 2: Style Refinement: The output from Stage 1 is further refined to achieve better style and texture matching with the target image. This stage uses the blended image from Stage 1 as input and optimizes it with respect to the target image's style and content, using similar content and style losses but with different input images.

Datasets

The paper does not explicitly mention specific datasets for training. Instead, it emphasizes that their algorithm does not rely on any training data and can generalize to any source and target images. The evaluation is performed on a variety of real-world images and stylized paintings.

Key Insights and Ideas

  • Differentiable Poisson Loss: By reformulating the Poisson blending objective as a differentiable loss function, the authors enable its integration with other deep learning losses (content and style), overcoming a key limitation of traditional Poisson blending.
  • Two-Stage Approach for Comprehensive Blending: Separating the problem into seamless boundary creation (Stage 1) and style refinement (Stage 2) allows for a more targeted and effective optimization process, leading to superior results.
  • Generalization without Training Data: The ability to perform high-quality blending without requiring paired training data is a significant advantage, making the method broadly applicable to diverse image types and scenarios.
  • Leveraging Deep Features for Style and Content: The use of deep convolutional networks (VGG) to extract features for content and style losses is crucial for achieving perceptually pleasing and stylistically consistent blends.

Key Results

The paper demonstrates strong qualitative results across various scenarios, including blending objects onto paintings and real-world images.

  • Ablation Studies: The ablation study (Figure 5) shows that each component of their method (Poisson gradient loss, content loss, style loss, and the two-stage approach) contributes significantly to the final result. Removing any component leads to noticeable artifacts or reduced quality.
  • Comparison with Baselines:
    • On paintings (Figure 6), their method produces more consistent style and texture while preserving source content, outperforming "copy-and-paste," "Poisson Blending," "Poisson Blending + Style Transfer," and "Style Transfer + Poisson Blending."
    • On real-world images (Figure 7), their method is compared against "Deep Harmonization," "Poisson Blending," and "GP-GAN." The proposed method achieves superior results in terms of blending boundary, texture, and color illumination.
  • User Studies: User studies (Figure 8) indicate that their method is preferred by users 80-90% of the time, with nine out of ten real-world image sets and eight out of ten painting-style image sets receiving the highest votes.
  • Runtime: The runtime for a 512x512 image is approximately 5 minutes on a single NVIDIA GTX 1080Ti.

Limitations and Future Work

  • Hyperparameter Tuning: The authors note that careful tuning of hyperparameters might yield even better results, suggesting that the current settings are not necessarily optimal.
  • Computational Cost: While efficient for a deep learning approach, the 5-minute runtime for a single image might still be a consideration for real-time applications.
  • Future Work: The paper suggests that their work could be applied in common image editing tasks and opens new possibilities for users to easily compose artistic works, implying potential for further exploration in creative image manipulation.

Summary updated

BibTeX
@inproceedings{zhang2020deep,
  author = {Lingzhi Zhang and Tarmily Wen and Jianbo Shi},
  title = {{Deep Image Blending}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure