Perceptual Adversarial Networks for Image-to-Image Transformation

Chaoyue Wang, Chang Xu, Chaohui Wang, Dacheng Tao

Abstract

In this paper, we propose a principled Perceptual Adversarial Networks (PAN) for image-to-image transformation tasks. Unlike existing application-specific algorithms, PAN provides a generic framework of learning mapping relationship between paired images, such as mapping a rainy image to its de-rained counterpart, object edges to its photo, semantic labels to a scenes image, etc. The proposed PAN consists of two feed-forward convolutional neural networks (CNNs), the image transformation network T and the discriminative network D. Through combining the generative adversarial loss and the proposed perceptual adversarial loss, these two networks can be trained alternately to solve image-to-image transformation tasks. Among them, the hidden layers and output of the discriminative network D are upgraded to continually and automatically discover the discrepancy between the transformed image and the corresponding ground-truth. Simultaneously, the image transformation network T is trained to minimize the discrepancy explored by the discriminative network D. Through the adversarial training process, the image transformation network T will continually narrow the gap between transformed images and ground-truth images. Experiments evaluated on several image-to-image transformation tasks (e.g., image de-raining, image inpainting, etc.) show that the proposed PAN outperforms many related state-of-the-art methods.

Introduction

Image-to-image transformation—such as turning a sketch into a photo or removing rain from an image—is a core challenge in computer vision. Traditional approaches often fall into two traps: pixel-wise losses (L1/L2) that tend to produce blurry results, and fixed perceptual losses (like VGG) that may not be optimal for task-specific textures and local geometry.

Perceptual Adversarial Networks (PAN) introduce a novel "perceptual adversarial loss" for these tasks. Unlike standard methods that rely on fixed, pre-trained networks to measure image similarity, PAN trains the hidden layers of its own discriminator to dynamically discover and penalize discrepancies between generated images and ground truth.

Overview of various image-to-image transformation tasks handled by the PAN framework, including de-raining, label-to-scene mapping, and inpainting.

The PAN Framework

The PAN framework consists of two primary components:

  • Image Transformation Network ($T$): An encoder-decoder architecture with skip connections (similar to a U-Net) that maps the input image to the target domain.
  • Discriminative Network ($D$): A convolutional classifier that distinguishes between real and transformed images.

Perceptual Adversarial Loss

Standard GANs only use the final output of the discriminator (the "Real vs. Fake" score). PAN goes deeper. It compares the feature maps of the hidden layers of $D$ for both the generated image and the ground truth.

  • The generator ($T$) tries to minimize the difference between these hidden features.
  • The discriminator ($D$) is updated to ensure its hidden layers are specifically tuned to find the differences between the generated output and the real image.

This creates a "moving target" for the generator, forcing it to match the ground truth across multiple levels of abstraction, from low-level textures to high-level structures.

Experiments and Datasets

The authors evaluated PAN across a wide variety of benchmarks:

  • ID-CGAN Dataset: Used for single-image de-raining.
  • ILSVRC’12 (ImageNet): A subset used for image inpainting (filling in missing central regions).
  • Cityscapes: For mapping semantic labels to street scenes and vice versa.
  • Pix2pix Datasets: Including Edges $\rightarrow$ Shoes/Handbags, Aerial photos $\rightarrow$ Maps, and Facades.

Key Results and Insights

PAN consistently outperformed state-of-the-art baselines (like pix2pix and ID-CGAN) both qualitatively and quantitatively. For instance, in de-raining, it achieved a PSNR of 23.35 dB and SSIM of 0.8303, significantly higher than the ID-CGAN baseline (22.91 dB / 0.8198).

Key insights include:

  • Dynamic vs. Static Metrics: By using the discriminator's hidden layers instead of a pre-trained VGG, the "perceptual metric" is learned specifically for the task at hand.
  • Layer Specialization: Lower layers of the discriminator focus on color and patch-to-patch consistency, while higher layers capture high-frequency details and structural information.
  • Generic Framework: PAN is a "plug-and-play" framework that works across diverse domains without changing the underlying architecture.

Conclusion

While PAN shows significant improvements in visual quality and artifact reduction, challenges remain, such as minor color distortions in certain tasks and sensitivity to hyperparameters. Future work suggests extending the perceptual adversarial loss to unpaired frameworks like CycleGAN for unsupervised image translation.

Summary updated

Abstract

In this paper, we propose a principled Perceptual Adversarial Networks (PAN) for image-to-image transformation tasks. Unlike existing application-specific algorithms, PAN provides a generic framework of learning mapping relationship between paired images (Fig. 1), such as mapping a rainy image to its de-rained counterpart, object edges to its photo, semantic labels to a scenes image, etc. The proposed PAN consists of two feed-forward convolutional neural networks (CNNs), the image transformation network T and the discriminative network D. Through combining the generative adversarial loss and the proposed perceptual adversarial loss, these two networks can be trained alternately to solve image-to-image transformation tasks. Among them, the hidden layers and output of the discriminative network D are upgraded to continually and automatically discover the discrepancy between the transformed image and the corresponding ground-truth. Simultaneously, the image transformation network T is trained to minimize the discrepancy explored by the discriminative network D. Through the adversarial training process, the image transformation network T will continually narrow the gap between transformed images and ground-truth images. Experiments evaluated on several image-to-image transformation tasks (e.g., image de-raining, image inpainting, etc.) show that the proposed PAN outperforms many related state-of-the-art methods.
BibTeX
@article{wang2018perceptual,
  author = {Chaoyue Wang and Chang Xu and Chaohui Wang and Dacheng Tao},
  title = {{Perceptual Adversarial Networks for Image-to-Image Transformation}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=1CNf7Q1JwaI},
}
Figure