Quality Aware Generative Adversarial Networks

Parimala Kancharla, Sumohana S. Channappayya

In the world of Generative Adversarial Networks (GANs), researchers have spent years perfecting the "cat-and-mouse" game between a Generator trying to create realistic images and a Discriminator trying to spot the fakes. While sophisticated metrics like SSIM or NIQE are frequently used to evaluate how well these models perform after training, they are rarely used to actually guide the training process itself.

In their NeurIPS 2019 paper, "Quality Aware Generative Adversarial Networks" (QAGANs), Parimala Kancharla and Sumohana S. Channappayya bridge this gap. They demonstrate that by baking human-centric image quality metrics directly into the GAN loss function, models can achieve significantly better visual results and more stable training.

The Disconnect Between Training and Evaluation

Despite their immense popularity, GANs are notoriously difficult to train. They frequently suffer from mode collapse (where the generator produces the same few images), convergence issues, and "noisy" visual outputs that lack the crispness of real photos.

Researchers typically tackle these problems using mathematical constraints, such as the Lipschitz continuity enforced in Wasserstein GANs (WGAN-GP). However, these constraints are often purely mathematical and do not explicitly account for what makes an image look "good" to the human eye. The authors identified a crucial gap: while we have excellent tools to measure image quality, they are rarely used as cost functions because they are mathematically unwieldy—often non-convex and difficult to differentiate.

Making GANs "Quality Aware"

To address this, the authors propose two primary methods to integrate objective Image Quality Assessment (IQA) metrics as regularizers within the GAN framework:

1. SSIM-based Regularization

The Structural Similarity Index (SSIM) is a gold standard for comparing a distorted image to a perfect reference image. Because standard SSIM is not a valid mathematical distance metric (it can yield negative values), the authors utilized a specific variant called $d^Q$. This transformation turns SSIM into a valid distance metric, which is then used to enforce a "Quality Aware" Lipschitz constraint on the discriminator.

2. NIQE-based Gradient Penalty

The Natural Image Quality Evaluator (NIQE) is a "no-reference" metric, meaning it can judge an image's quality without needing a perfect baseline. It evaluates the statistical signature of an image, specifically looking at Mean Subtracted Contrast Normalized (MSCN) coefficients.

The authors made a fascinating discovery: the gradients of the discriminator also follow these natural statistical signatures. They proposed a novel penalty function that punishes the discriminator if its gradients deviate from the statistical "fingerprint" of real-world natural images.

Comparison of MSCN coefficient distributions for natural scenes, spatial gradients, and discriminator gradients.

These regularizers were integrated into the WGAN-GP framework, creating a hybrid loss function that balances traditional adversarial training with explicit quality constraints.

Key Insights

  • The Statistical Signature of Gradients: Perhaps the most surprising finding is that the discriminator's gradients are not just random noise. When trained on natural images, they inherit the same Gaussian-like statistical properties as the images themselves.
  • Bridging the Gap: By using IQA metrics during training, the model essentially "pre-tests" itself against the very metrics researchers use to judge it at the end.
  • Stability through Structure: Traditional gradient penalties only care about the norm (length) of the gradient. QAGANs care about the structure and correlation of the gradient, leading to more meaningful updates for the generator.

State-of-the-Art Results

The researchers evaluated QAGANs on standard computer vision benchmarks (CIFAR-10, STL-10, and CelebA), measuring success using the Inception Score (IS) and the Frechet Inception Distance (FID).

  • Massive FID Improvements: On the CIFAR-10 dataset, the standard WGAN-GP achieved an FID of 40.2. The SSIM-based QAGAN slashed that to 13.91, and the NIQE-based version reached an impressive 12.46.
  • CelebA Performance: For face generation, the SSIM-based QAGAN achieved an FID of 6.42, nearly doubling the performance of the baseline WGAN-GP (12.89).
  • Scaling Up: When applied to Progressive GANs (PGGAN) at 256x256 resolution, adding QAGAN regularizers dropped the FID from 62.86 to 40.83, proving these methods scale well to high-resolution image synthesis.
Sample generated images demonstrating the high visual quality achieved by the QAGAN framework.

Limitations and Future Work

While highly effective, the approach has a few limitations. The NIQE-based regularizer showed inconsistent performance on very small images (like the 32x32 CIFAR-10 dataset), likely due to the difficulty of estimating statistical parameters from so few pixels. Additionally, calculating MSCN coefficients and Multivariate Gaussian distances adds computational complexity to the training loop.

However, this work paves the way for a new paradigm in generative modeling. The authors suggest that the plethora of excellent quality assessment metrics available in the image processing community could soon be translated into even more powerful GAN regularizers.

Summary updated

Abstract

Generative Adversarial Networks (GANs) have become a very popular tool for im- plicitly learning high-dimensional probability distributions. Several improvements have been made to the original GAN formulation to address some of its shortcom- ings like mode collapse, convergence issues, entanglement, poor visual quality etc. While a significant effort has been directed towards improving the visual quality of images generated by GANs, it is rather surprising that objective image quality metrics have neither been employed as cost functions nor as regularizers in GAN objective functions. In this work, we show how a distance metric that is a variant of the Structural SIMilarity (SSIM) index (a popular full-reference image quality assessment algorithm), and a novel quality aware discriminator gradient penalty function that is inspired by the Natural Image Quality Evaluator (NIQE, a popular no-reference image quality assessment algorithm) can each be used as excellent regularizers for GAN objective functions. Specifically, we demonstrate state-of- the-art performance using the Wasserstein GAN gradient penalty (WGAN-GP) framework over CIFAR-10, STL10 and CelebA datasets.
BibTeX
@inproceedings{kancharla2019quality,
  author = {Parimala Kancharla and Sumohana S. Channappayya},
  title = {{Quality Aware Generative Adversarial Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=nbPXlNoVPo},
}
Figure