IEGAN: Multi-purpose Perceptual Quality Image Enhancement Using Generative Adversarial Network

Soumya Shubhra Ghosh, Yang Hua, Sankha Subhra Mukherjee, Neil Martin Robertson

TL;DR

IEGAN is a novel generative adversarial network that tackles image super-resolution and compression artifact removal simultaneously within a single end-to-end framework. By introducing an explicit edge loss and an autoencoder-based discriminator, it restores sharp, photo-realistic color images where traditional sequential pipelines fail.

What Problem Does This Paper Address?

Real-world digital images—whether scraped from social media or captured by surveillance cameras—frequently suffer from two degradation types simultaneously: low resolution and heavy compression artifacts (such as JPEG blocking and blurriness).

Traditionally, the computer vision community treats Super-Resolution (SR) and Artifact Removal (AR) as separate problems. When faced with an image suffering from both, practitioners typically stack models sequentially (e.g., running an AR network followed by an SR network). However, this approach compounds errors, smooths out fine textures, and introduces unnatural noise patterns. Furthermore, most existing AR algorithms operate solely on the monochrome luminance (Y) channel, ignoring color artifacts.

Finally, standard models optimize for pixel-wise reconstruction errors like Mean Squared Error (MSE). While this maximizes traditional metrics like Peak Signal-to-Noise Ratio (PSNR), it yields perceptually disappointing, blurry results that lack high-frequency details.

Methods

The authors introduce the Image Enhancement Generative Adversarial Network (IEGAN) to handle both tasks at once in full RGB color space.

Low-Res, Compressed Image ---> [ U-Net Generator ] ---> High-Res, Clean Image
                                       |
                                       v
                        [ Autoencoder Discriminator ]
                                       |
                Calculates Tri-Partite Perceptual Loss
                (VGG Feature + Canny Edge + Recon Loss)

1. Network Architecture

  • Generator ($G_{\theta_G}$): Instead of relying entirely on standard residual blocks, the authors implement a U-Net-like architecture. Skip connections concatenate early layers directly with later layers, allowing low-level structural information to flow across the network. Pixel-shuffle layers are appended at the end to handle the $4\times$ upscaling.
  • Discriminator ($D_{\theta_D}$): Instead of a traditional binary classifier that outputs a single "real vs. fake" scalar, IEGAN utilizes an autoencoder-like discriminator. It attempts to reconstruct the input image, passing dense semantic feedback to the generator based on the distribution of reconstruction losses.

2. Tri-Partite Perceptual Loss Function

To enforce photo-realism, the final loss function combines three distinct components:

  1. Feature Loss: Measures Euclidean distance between feature maps in a pre-trained VGG19 network to preserve high-level textures.
  2. Edge Loss: Explicitly penalizes blurry boundaries using an edge detection function ($\Theta$). The authors tested both classical Canny filters and deep Holistically-nested Edge Detection (HED).
  3. Reconstruction Loss: Derived from the autoencoder discriminator, balancing the loss distributions of real and generated images.

Datasets

  • Training: A random sample of 60,000 images from the ImageNet dataset.
  • Evaluation: Standard public benchmarks including LIVE1 (for AR), and Set14 and BSD100 (for SR).
  • New Benchmark: The authors introduce World100, a dataset containing 100 high-resolution color photographs featuring complex textures, sharp edges, and color gradients specifically designed to evaluate simultaneous AR+SR performance.

Key Insights and Ideas

  • The Order-of-Operations Trap: Stacking independent models (AR then SR, or SR then AR) creates a fundamental mismatch. A joint end-to-end network is significantly better at navigating the natural image manifold because it optimizes for both degradations at the same time.
  • Simple Edges Win: Surprisingly, the classical, lightweight Canny edge detector performed on par with the complex, deep learning-based HED model when used in the edge loss component. This allowed the authors to opt for the faster, less resource-intensive Canny method without sacrificing sharpness.
  • Rethinking the Discriminator: Moving away from a binary "real/fake" discriminator to an autoencoder structure provides the generator with richer, pixel-aligned semantic gradients, which is critical for restoring fine details.

Key Results

IEGAN was evaluated using perceptual metrics like Gradient Magnitude Similarity Deviation (GMSD) and Haar wavelet-based Perceptual Similarity Index (HaarPSI), which correlate better with human vision than PSNR.

  • Simultaneous AR+SR (LIVE1): IEGAN achieved a GMSD of 0.1404 and a HaarPSI of 0.5504, drastically outperforming sequential pipelines like ARCNN+SRGAN (GMSD: 0.1980, HaarPSI: 0.4112).
  • Simultaneous AR+SR (World100): On the authors' high-resolution dataset, IEGAN maintained the best perceptual fidelity with a GMSD of 0.1009, compared to 0.1668 for ARCNN+SRGAN.
  • Visual Quality: Qualitatively, IEGAN successfully eliminates blocky JPEG artifacts and reconstructs sharp, believable textures in full RGB color, whereas competing methods leave images looking blotchy or overly smooth.

Limitations and Future Work

The primary limitation of IEGAN lies in handling large, gentle color gradients (such as clear skies or auroras).

Because the network is trained on relatively small cropped sub-images ($128 \times 128$), its receptive field cannot grasp massive, gradual color shifts present in 2000+ pixel images. As a result, all tested algorithms—including IEGAN—fail to perfectly recreate these smooth transitions, sometimes introducing subtle banding artifacts. The authors suggest that expanding the network's global context or adjusting training patch scales will be an important direction for future work.

Summary updated

Abstract

Despite the breakthroughs in quality of image enhancement, an end-to-end solution for simultaneous recovery of the finer texture details and sharpness for degraded images with low resolution is still unsolved. Some existing approaches focus on minimizing the pixel-wise reconstruction error which results in a high peak signal-to-noise ratio. The enhanced images fail to provide high-frequency details and are perceptually unsatisfying, i.e., they fail to match the quality expected in a photo-realistic image. In this paper, we present Image Enhancement Generative Adversarial Network (IEGAN), a versatile framework capable of inferring photo-realistic natural images for both artifact removal and super-resolution simultaneously. Moreover, we propose a new loss function consisting of a combination of reconstruction loss, feature loss and an edge loss counterpart. The feature loss helps to push the output image to the natural image manifold and the edge loss preserves the sharpness of the output image. The reconstruction loss provides low-level semantic information to the generator regarding the quality of the generated images compared to the original. Our approach has been experimentally proven to recover photo-realistic textures from heavily compressed low-resolution images on public benchmarks and our proposed high-resolution World100 dataset.
BibTeX
@misc{ghosh2018iegan,
  author = {Soumya Shubhra Ghosh and Yang Hua and Sankha Subhra Mukherjee and Neil Martin Robertson},
  title = {{IEGAN: Multi-purpose Perceptual Quality Image Enhancement Using Generative Adversarial Network}},
  year = {2018},
  eprint = {1811.09134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.09134},
}

Loading related content...

Figure