DCIL: Deep Contextual Internal Learning for Image Restoration and Image Retargeting

Indra Deep Mastan, Shanmuganathan RamanWACV 2020

TL;DR

This paper introduces Deep Contextual Internal Learning (DCIL), a novel unsupervised framework for image resizing tasks like restoration and retargeting. By combining network structure as an implicit image prior with a generalized loss function, DCIL achieves state-of-the-art performance without relying on training datasets.

What Problem Does This Paper Address?

Deep learning has revolutionized image processing, but supervised methods often require large, labeled datasets. This can be problematic when data is scarce, expensive to acquire, or when dealing with complex, unknown degradations. Unsupervised methods offer a solution by learning from a single image, but they face challenges, particularly in tasks like image retargeting where the source and target images lack spatial correspondence. Existing unsupervised approaches like Deep Image Prior (DIP) and Zero-Shot Super-Resolution (ZSSR) primarily rely on pixel-to-pixel losses, limiting their applicability to non-aligned data. Contextual loss has been proposed for non-aligned data, but it's not entirely independent of training samples. This paper aims to bridge this gap by proposing a unified, unsupervised framework that handles both aligned (restoration) and non-aligned (retargeting) image resizing tasks effectively.

Methods

The core of DCIL is a generator-discriminator network architecture, inspired by Generative Adversarial Networks (GANs), but crucially, it operates without training data. The "Deep Contextual Internal Learning" (DCIL) framework leverages two key components:

  • Network Construction: The network's structure itself acts as an implicit image prior. The authors formalize this by defining a network F as a combination of layers ($\Phi$), skip connections ($S$), cascading inputs ($C$), and residual blocks ($R$). They emphasize the importance of skip connections and residual blocks for improving performance.
  • Generalized Loss Function: DCIL employs a total loss function that combines three types of comparisons between the source image ($x$) and the generated target image ($y$):
  • Contextual Loss ($L_{CL}$): This loss, computed using features from a pre-trained VGG19 network, measures contextual similarity between the source and target images. It's designed to work with non-aligned data and enhances feature learning at the scale of the target image.
  • Adversarial Loss ($L_{GAN}$): This loss, inspired by GANs, uses a multi-scale patch discriminator to match the patch distribution of the generated image to that of natural images. This helps in generating realistic image patches and is crucial for tasks like retargeting.
  • Reconstruction Loss ($L_R$): This loss ensures the preservation of global image features. For image restoration, it's a pixel-to-pixel comparison between the generated image and the source. For image retargeting, it uses a cycle-consistent approach, comparing the source image to an inverse mapping of the generated output.

The framework is modular, allowing for task-specific loss function adjustments. For instance, Denoising-Super-Resolution (DSR) incorporates a Total Variation (TV) regularization term to reduce noise.

Datasets

The paper evaluates its framework on standard datasets commonly used for image restoration and retargeting tasks. Specific datasets mentioned include:

  • BSD100: Used for Super-Resolution (SR) and Denoising-Super-Resolution (DSR) evaluations.
  • Set5 and Set14: Also used for SR evaluations.

The authors do not introduce new datasets but rather demonstrate the effectiveness of their unsupervised approach on existing benchmarks.

Key Insights and Ideas

  • Network Structure as Prior: The paper strongly advocates for the idea that the architecture of a neural network can inherently act as a powerful image prior, eliminating the need for explicit handcrafted priors or large training datasets.
  • Unified Framework for Diverse Tasks: DCIL successfully unifies image restoration (aligned data) and image retargeting (non-aligned data) under a single unsupervised framework, demonstrating its versatility.
  • Contextual Loss for Non-Aligned Data: The effective use of contextual loss is highlighted as a key enabler for handling image retargeting, where direct pixel correspondence is absent.
  • Adversarial Learning for Realistic Patches: The integration of adversarial loss is shown to be crucial for generating realistic image patches and maintaining object distributions in retargeting.
  • Modularity for Task Specialization: The framework's modular design, particularly in its loss functions, allows for adaptation to specific tasks like DSR by incorporating noise reduction mechanisms.

Key Results

The paper presents strong quantitative and qualitative results across several image resizing tasks:

  • Denoising-Super-Resolution (DSR): DCIL outperforms existing methods like CL [18] and DIP [26] in 2x DSR with significant noise (σ=100), producing cleaner images with better perceptual quality (Figure 3).
  • Super-Resolution (SR): For 4x SR on BSD100, DCIL achieves competitive SSIM scores, comparable to DIP [26] and other state-of-the-art methods. While pixel-to-pixel methods might yield higher PSNR, DCIL focuses on perceptual quality.
  • Image Retargeting: DCIL demonstrates superior performance in preserving object structure and alignment compared to methods like Seam-Carving (SC) [5] (Figures 5 & 6). It also performs comparably to InGAN [22] in challenging cases (Figure 8). A user study showed DCIL received 65% of votes for perceptually better images, significantly outperforming SC [5] (35%).
  • Image Inpainting: The framework also shows promise for image inpainting tasks, leveraging its self-similarity prior (Figure 9).

Limitations and Future Work

  • Contextual Understanding: The authors acknowledge that the framework's limitations stem from the inherent lack of contextual understanding when learning from a single image. This necessitates careful design of application-specific networks.
  • Perceptual Quality in Object Replication: While DCIL excels at object partitioning in retargeting, the perceptual quality in scenarios involving object replication could be further improved.
  • Hyper-parameter Tuning: The paper notes that achieving optimal results, especially with GANs and contextual loss, often requires extensive hyper-parameter search.
  • Future Work: The authors suggest investigating the image statistics captured by DCIL for other single-image applications, such as image inpainting, and exploring its potential for more challenging image resizing scenarios. They also mention the possibility of further improving performance through hyper-parameter tuning.

Summary updated

BibTeX
@inproceedings{mastan2020dcil,
  author = {Indra Deep Mastan and Shanmuganathan Raman},
  title = {{DCIL: Deep Contextual Internal Learning for Image Restoration and Image Retargeting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure