Perceptual Adversarial Networks for Image-to-Image Transformation

Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao

TL;DR

Perceptual Adversarial Networks (PAN) introduce a novel "perceptual adversarial loss" for image-to-image transformation tasks. Unlike standard methods that rely on fixed, pre-trained networks (like VGG) to measure image similarity, PAN trains the hidden layers of its own discriminator to dynamically discover and penalize discrepancies between generated images and ground truth.

What Problem Does This Paper Address?

Image-to-image transformation—such as turning a sketch into a photo or removing rain from an image—is a core challenge in computer vision. Traditional approaches often fall into two traps:

  1. Pixel-wise losses (L1/L2): These tend to produce blurry results because they average out possible pixel values.
  2. Fixed Perceptual losses: Many models use a pre-trained VGG network to compare "high-level" features. However, VGG is trained for object classification, meaning its features might not be optimal for tasks like de-raining or map generation where texture and local geometry matter more than "is this a dog?"

The authors aim to create a generic framework that learns a task-specific perceptual metric that evolves alongside the image generator.

Methods

The PAN framework consists of two primary components:

  • Image Transformation Network ($T$): An encoder-decoder architecture with skip connections (similar to a U-Net) that maps the input image to the target domain.
  • Discriminative Network ($D$): A convolutional classifier that distinguishes between real and transformed images.

The Secret Sauce: Perceptual Adversarial Loss Standard GANs only use the final output of the discriminator (the "Real vs. Fake" score). PAN goes deeper. It compares the feature maps of the hidden layers of $D$ for both the generated image and the ground truth.

  • The generator ($T$) tries to minimize the difference between these hidden features.
  • The discriminator ($D$) is updated to ensure its hidden layers are specifically tuned to find the differences between the generated output and the real image.

This creates a "moving target" for the generator, forcing it to match the ground truth across multiple levels of abstraction (from low-level textures to high-level structures).

Datasets

The authors evaluated PAN across a wide variety of benchmarks:

  • ID-CGAN Dataset: Used for single-image de-raining.
  • ILSVRC’12 (ImageNet): A subset used for image inpainting (filling in missing central regions).
  • Cityscapes: For mapping semantic labels to street scenes and vice versa.
  • Pix2pix Datasets: Including Edges $\rightarrow$ Shoes/Handbags, Aerial photos $\rightarrow$ Maps, and Facades.

Key Insights and Ideas

  • Dynamic vs. Static Metrics: By using the discriminator's hidden layers instead of a pre-trained VGG, the "perceptual metric" is learned specifically for the task at hand. If the model is de-raining, the discriminator learns to look for rain streaks; if it's generating maps, it learns to look for road boundaries.
  • Layer Specialization: The authors found that lower layers of the discriminator focus on color and patch-to-patch consistency, while higher layers capture high-frequency details and structural information.
  • Generic Framework: Unlike application-specific algorithms, PAN is a "plug-and-play" framework that works across diverse domains without changing the underlying architecture.

Key Results

PAN consistently outperformed state-of-the-art baselines (like pix2pix and ID-CGAN) both qualitatively and quantitatively:

  • De-raining: Achieved a PSNR of 23.35 dB and SSIM of 0.8303, significantly higher than the ID-CGAN baseline (22.91 dB / 0.8198).
  • Inpainting: Produced sharper details and better semantic understanding of the surroundings compared to the Context-Encoder (CE) method.
  • Visual Quality: Qualitative results show that PAN produces images with fewer artifacts and better color preservation than methods using standard perceptual losses.

Limitations and Future Work

  • Color Distortion: While improved, the authors noted that some tasks (like de-snowing) still exhibit minor color shifts compared to the original input.
  • Hyperparameter Sensitivity: The balance between the standard GAN loss and the perceptual adversarial loss (controlled by $\theta$) requires careful tuning for different tasks.
  • Unpaired Data: While the paper primarily focuses on paired training data, the authors briefly demonstrated that the perceptual adversarial loss could be extended to unpaired frameworks like CycleGAN, suggesting a promising direction for future research in unsupervised image translation.

Summary updated

Abstract

In this paper, we propose perceptual adversarial networks (PANs) for image-to-image transformations. Different from existing application driven algorithms, PAN provides a generic framework of learning to map from input images to desired images (Fig. 1), such as a rainy image to its de-rained counterpart, object edges to photos, and semantic labels to a scenes image. The proposed PAN consists of two feed-forward convolutional neural networks: the image transformation network T and the discriminative network D. Besides the generative adversarial loss widely used in GANs, we propose the perceptual adversarial loss, which undergoes an adversarial training process between the image transformation network T and the hidden layers of the discriminative network D. The hidden layers and the output of the discriminative network D are upgraded to constantly and automatically discover the discrepancy between the transformed image and the corresponding ground truth, while the image transformation network T is trained to minimize the discrepancy explored by the discriminative network D. Through integrating the generative adversarial loss and the perceptual adversarial loss, D and T can be trained alternately to solve image-to-image transformation tasks. Experiments evaluated on several image-to-image transformation tasks (e.g., image deraining and image inpainting) demonstrate the effectiveness of the proposed PAN and its advantages over many existing works.
BibTeX
@article{wang2018perceptual,
  author = {Chaoyue Wang and Chang Xu and Chaohui Wang and Dacheng Tao},
  title = {{Perceptual Adversarial Networks for Image-to-Image Transformation}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=1CNf7Q1JwaI},
}

Loading related content...

Figure