Learning Generalizable Perceptual Representations for Data-Efficient No-Reference Image Quality Assessment

Suhas Srinath, Shankhanil Mitra, Shika Rao, Rajiv SoundararajanWACV 2024

TL;DR

This paper introduces GRepQ, a novel framework for data-efficient no-reference image quality assessment (NR-IQA). GRepQ learns generalizable low-level and high-level image features using unsupervised contrastive learning, significantly reducing the need for human annotations and enabling accurate quality predictions even in zero-shot settings.

What Problem Does This Paper Address?

Assessing image quality without a reference image (NR-IQA) is crucial for user experience across various applications. However, current state-of-the-art NR-IQA methods heavily rely on large datasets of human-annotated images for training. This is a significant bottleneck, as creating such datasets is time-consuming and expensive, especially when dealing with evolving image distortions. The paper aims to address this by developing methods that can learn generalizable quality representations from limited data or even without any human labels, making NR-IQA more practical and scalable.

Methods

GRepQ employs a two-pronged approach to learn generalizable image quality representations:

  • Low-Level Representation Model (Quality-Aware Contrastive Loss): This component focuses on learning features that capture local, distortion-agnostic quality variations. It uses a novel contrastive loss that assigns "soft" weights to positive and negative training pairs based on a perceptual similarity measure between distorted versions of an image. This approach allows the model to learn representations that are invariant to distortion types, unlike previous methods that might pull apart features from images with different distortions, even if they have similar perceptual quality.
  • High-Level Representation Model (Unsupervised Vision-Language Model Fine-tuning): This component leverages the power of pre-trained vision-language models, specifically CLIP. Instead of direct fine-tuning on IQA data, GRepQ uses an unsupervised group-contrastive learning strategy. It segregates images into "high-quality" and "low-quality" groups using antonym text prompts (e.g., "good photo" vs. "bad photo") and trains the CLIP image encoder to align representations within groups and separate them across groups. This allows the model to capture semantic quality information without explicit human labels.

These two sets of features are then combined. For data-efficient IQA, a simple regressor is trained with very few labeled samples. For zero-shot IQA, the features are used directly to predict quality without any further training on the target dataset.

Datasets

  • Training: The low-level and high-level feature encoders were trained on the FLIVE dataset (5000 real-world images with authentic distortions). Notably, no human annotations were used during this training phase.
  • Evaluation: The framework was evaluated on a variety of datasets to demonstrate its generalizability:
  • In-the-wild datasets: CLIVE (1,162 images from mobile devices) and KonIQ-10K (10,073 in-the-wild images).
  • Synthetic/processed datasets: LIVE-IQA (779 images with blur, compression, noise, fast-fading), CSIQ (866 images with blur, contrast, JPEG compression), and PIPAL (23,200 images with 40 different distortions, including GAN artifacts).

Key Insights and Ideas

  • Distortion-Agnostic Low-Level Features: The quality-aware contrastive loss is a key innovation. By weighting similarity based on perceptual quality rather than just distortion type, the model learns features that generalize better across unseen distortions.
  • Leveraging Vision-Language Models for IQA: Fine-tuning CLIP using unsupervised group-contrastive learning with antonym prompts is an effective way to extract high-level semantic quality information without requiring labeled IQA data. This bridges the gap between general vision-language understanding and specific IQA tasks.
  • Complementarity of Low and High-Level Features: The paper demonstrates that combining both low-level (distortion-focused) and high-level (semantic context-focused) features leads to superior performance, especially in challenging scenarios where one type of feature might fail.
  • Data Efficiency is Achievable: The framework shows that accurate NR-IQA can be achieved with very few labeled samples, making it practical for real-world applications where data annotation is a bottleneck.
  • Zero-Shot Performance: The ability to predict image quality without any training on the target dataset is a significant advantage, showcasing the strong generalization capabilities of the learned representations.

Key Results

  • Data-Efficient Setting: GRepQD consistently outperforms other state-of-the-art NR-IQA methods on multiple datasets across different data regimes (50, 100, 200 samples), often surpassing even end-to-end trained models.
  • Zero-Shot Setting: GRepQz achieves state-of-the-art performance on three out of five evaluated datasets, including the challenging PIPAL dataset, showing a significant improvement (e.g., 44% SRCC improvement over the second-best method on PIPAL).
  • Cross-Database Experiments: The learned features demonstrate strong performance in cross-database evaluation, indicating their robustness and generalizability.
  • Perceptual Similarity Measure Impact: FSIM was found to be the most effective perceptual similarity measure for training the low-level encoder, outperforming others like SSIM and LPIPS.

Limitations and Future Work

  • Low-Level Model on In-the-Wild Data: In the low-data setting, the low-level model doesn't perform as well as the high-level model on in-the-wild datasets. This is attributed to its suitability for capturing distortion levels rather than content, making it more beneficial for synthetic datasets.
  • Prompt Engineering for High-Level Model: The high-level model uses fixed text prompts. Future work could involve prompt engineering or tuning to further enhance its performance.
  • Future Directions: The authors suggest that future advances in self-supervised learning and quality-specific prompt engineering could further improve the generalizability of models for data-efficient NR-IQA.

Summary updated

BibTeX
@inproceedings{srinath2024learning,
  author = {Suhas Srinath and Shankhanil Mitra and Shika Rao and Rajiv Soundararajan},
  title = {{Learning Generalizable Perceptual Representations for Data-Efficient No-Reference Image Quality Assessment}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure