SiENet: Siamese Expansion Network for Image Extrapolation

Xiaofeng Zhang, Feng Chen, Cailing Wang, Ming Tao, Guo-Ping JiangIEEE Signal Processing Letters 2020 · DOI

TL;DR

This paper introduces SiENet, a novel two-stage Siamese adversarial network for image extrapolation (outpainting). By incorporating an "adaptive filling convolution" into the encoder and a "siamese adversarial mechanism," SiENet effectively balances the inferring burden between the encoder and decoder, leading to more realistic and coherent extrapolated images.

What Problem Does This Paper Address?

Image extrapolation, also known as image outpainting, is the task of generating new content beyond the original boundaries of an image. Unlike image inpainting, which fills in missing regions within an image, extrapolation deals with generating content that is entirely outside the original image frame. This presents a unique challenge because the network has significantly less contextual information to infer a much larger unknown area. Traditional encoder-decoder architectures, commonly used in image generation tasks, struggle with extrapolation because they place the entire burden of inferring the unknown content on the decoder. This can lead to a weak encoder that fails to capture sufficient information and a decoder that is overwhelmed, resulting in unrealistic or incoherent extrapolations. Existing methods often adapt inpainting techniques, which are not ideally suited for the distinct challenges of extrapolation.

Methods

SiENet tackles the problem of image extrapolation with a novel two-stage architecture and two key innovations:

  • Adaptive Filling Convolution: This new convolution type is designed to specifically focus on inferring pixels in unknown areas. It works by adaptively balancing the influence of known neighboring pixels with a learned prediction, allowing the encoder to capture context and predict unknown content more effectively. This convolution is inserted into the encoder of the network, alleviating the burden on the decoder.
  • Siamese Adversarial Mechanism: This mechanism introduces prior knowledge into the network. It uses a siamese loss to encourage the features of the original (covered) image and the ground truth (uncovered) image to be similar in a learned subspace. This helps the encoder learn the distribution of real image features, reinforcing its inferring ability for long-range content. Additionally, adversarial discriminators are used in each stage to ensure the generated predictions are realistic.

The overall architecture is a two-stage pipeline:

  1. Structure Generator: This stage generates a smooth structure for the extrapolated region.
  2. Content Generator: This stage takes the original image, the filling map, and the generated structure as input to produce the final extrapolated image.

The loss function is a combination of several components: distance loss (for structure), adversarial loss, perceptual loss, style loss, and the novel siamese loss.

Datasets

The paper evaluates SiENet on four datasets:

  • Cityscapes: A dataset of urban street scenes.
  • Paris Street View: A dataset of street scenes in Paris.
  • Beach: A dataset of beach scenes.
  • Scenery: A dataset of natural scenery.

These datasets cover both street and nature scenarios, demonstrating the method's robustness.

Key Insights and Ideas

  • Encoder Empowerment is Crucial for Extrapolation: The paper strongly argues that relying solely on the decoder for extrapolation is insufficient. Empowering the encoder with specialized convolutions like the "adaptive filling convolution" is key to capturing and inferring the necessary information from limited context.
  • Siamese Learning for Prior Knowledge: The siamese adversarial mechanism is a clever way to inject prior knowledge about real image distributions into the encoder. By forcing the encoder to learn similar feature representations for both real and generated content, it improves its ability to predict plausible unknown regions.
  • Balancing the Burden: The two-stage approach combined with the adaptive filling convolution and siamese adversarial mechanism effectively distributes the inferring burden between the encoder and decoder, leading to a more stable and effective generation process.

Key Results

SiENet achieves state-of-the-art performance on several datasets for both two-direction and single-direction outpainting.

  • Quantitative Improvements: The paper reports significant improvements in SSIM and PSNR metrics compared to existing methods like Image-Outpainting, Outpainting-srn, Edge-Connect, and NS-Outpainting. For instance, on the Cityscapes dataset, SiENet achieves an SSIM of 0.7832 and a PSNR of 24.8213, outperforming previous methods.
  • Qualitative Results: Visual comparisons show that SiENet generates more coherent, realistic, and visually pleasing extrapolated images that are more similar to the ground truth than those produced by competing methods. The generated images exhibit better detail and consistency with the original content.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of a two-stage network and adversarial training generally implies a higher computational cost for training and inference compared to simpler models.
  • Future Work: The paper suggests that the proposed architecture is flexible for both two-direction and single-direction outpainting. Future work could explore further improvements in generating finer details or handling more complex scene compositions. The authors also mention releasing their code, which will facilitate further research in this area.

Summary updated

BibTeX
@article{zhang2020sienet,
  author = {Xiaofeng Zhang and Feng Chen and Cailing Wang and Ming Tao and Guo-Ping Jiang},
  title = {{SiENet: Siamese Expansion Network for Image Extrapolation}},
  journal = {IEEE Signal Processing Letters},
  year = {2020},
  doi = {10.1109/lsp.2020.3019705},
}
Figure