Opinion Unaware Image Quality Assessment via Adversarial Convolutional Variational Autoencoder

Ankit Shukla, Avinash Upadhyay, Swati Bhugra, Manoj SharmaWACV 2024

TL;DR

This paper introduces a novel unsupervised approach to Image Quality Assessment (IQA) that doesn't require subjective human scores (MOS). By training a Generative Adversarial Network (GAN) on pristine images, the model learns a latent representation of "good" image quality. Deviations from this learned distribution in test images are then used to predict their quality, achieving state-of-the-art results on various datasets.

What Problem Does This Paper Address?

Assessing image quality is crucial for many computer vision tasks, but it's challenging when you don't have a pristine reference image to compare against. Traditional "blind" image quality assessment (BIQA) methods often rely on subjective Mean Opinion Scores (MOS) collected from human observers. However, obtaining these scores is time-consuming and expensive, limiting the applicability of these "opinion-aware" methods. While unsupervised BIQA methods exist, they often still indirectly depend on MOS or are limited to specific distortion types. This paper aims to bridge this gap by developing an unsupervised BIQA method that learns directly from pristine images, making it more generalizable and practical for real-world scenarios.

Methods

The core of the proposed method is an Adversarial Convolutional Variational Autoencoder (CVAE-GAN) trained exclusively on pristine images. This framework has two main components:

  • CVAE (Convolutional Variational Autoencoder): This part learns to reconstruct pristine images. The encoder maps an input image to a low-dimensional "latent code" that follows a probabilistic distribution. The decoder then uses this code to generate a reconstructed image. The goal is to learn a rich, latent representation of high-quality images.
  • Discriminator: This network tries to distinguish between real pristine images and the reconstructed images produced by the CVAE. This adversarial training pushes the CVAE to generate highly realistic reconstructions.

The training process uses a combination of losses:

  • Kullback-Leibler Divergence (KLD) Loss: Ensures the latent representation of pristine images conforms to a standard normal distribution.
  • Perceptual Loss (VGG Loss): Uses features from a pre-trained VGG network to ensure the reconstructed images are perceptually similar to the originals.
  • Discriminator Loss: Drives the adversarial training, forcing the CVAE to produce high-quality, content-agnostic reconstructions.

For testing (unsupervised BIQA), the trained encoder's weights are used. The latent representation of a test image is extracted, and its deviation from the learned pristine distribution is measured using a "latent distance metric." This deviation serves as a pseudo-indicator of image degradation.

Datasets

  • Training Data: The model was trained on a large collection of pristine images from the KADIS dataset (140K images) and the Waterloo Exploration dataset (4744 images).
  • Evaluation Datasets: The proposed framework was evaluated on five benchmark datasets:
  • LIVE [44]: Contains synthetic distortions (JPEG compression, JPEG2000 compression, Gaussian blur, Gaussian noise, fast fading).
  • KADID-10k [19]: Contains synthetic distortions with 25 types at five severity levels.
  • CLIVE [8]: Contains authentic distortions.
  • FLIVE [43]: Contains authentic distortions.
  • KonIQ-10k [15]: Contains authentic distortions, rated by human observers.
  • SPAQ [7]: Contains authentic distortions from smartphone-captured images.

Key Insights and Ideas

  • Learning "Good" from Pristine: The core idea that a GAN can learn the distribution of "good" image quality solely from pristine examples is powerful. This bypasses the need for explicit human labels.
  • Latent Space as Quality Indicator: The t-SNE visualizations clearly show that distorted images occupy different regions in the learned latent space compared to pristine images. This separation is the foundation for unsupervised quality assessment.
  • Content-Agnostic Quality: By training on diverse pristine images and using a combination of losses (especially the discriminator loss), the model learns features that are less tied to specific image content and more to general quality degradation.
  • Generalization Across Distortions: The framework's ability to perform well across both synthetic and authentic distortions, as demonstrated by its performance on datasets like LIVE and KonIQ-10k, highlights its robustness.

Key Results

  • State-of-the-Art Performance: The proposed framework achieved state-of-the-art results on multiple benchmark datasets, outperforming other unsupervised BIQA methods and showing comparable performance to supervised methods.
  • Quantitative Metrics: The paper reports high SRCC and PLCC scores across various datasets, indicating strong correlation with human MOS. For example, on the KonIQ-10k dataset, the proposed method achieved an SRCC of 0.921 and PLCC of 0.931.
  • Ablation Studies:
    • Using VGG loss and KLD loss significantly improved performance over MSE loss and KLD loss, demonstrating the importance of perceptual features.
  • Adding the discriminator loss further enhanced performance, leading to more realistic reconstructions and better quality assessment.
  • The latent distance metric was found to be the most effective quality metric among those tested.

Limitations and Future Work

  • Reliance on Pristine Data: While the method doesn't need MOS, it still requires a substantial dataset of high-quality pristine images for training. The availability and diversity of this pristine data can influence performance.
  • Computational Cost: Training GANs can be computationally intensive.
  • Future Work: The authors suggest exploring the use of this framework for distortion identification and potentially extending it to video quality assessment. Further investigation into more efficient training strategies and exploring different GAN architectures could also be beneficial.

Summary updated

BibTeX
@inproceedings{shukla2024opinion,
  author = {Ankit Shukla and Avinash Upadhyay and Swati Bhugra and Manoj Sharma},
  title = {{Opinion Unaware Image Quality Assessment via Adversarial Convolutional Variational Autoencoder}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure