Transformer-Based No-Reference Image Quality Assessment via Supervised Contrastive Learning

Jinsong Shi, Pan Gao, Jie Qin

Abstract

Image Quality Assessment (IQA) has long been a research hotspot in the field of image processing, especially No-Reference Image Quality Assessment (NR-IQA). Due to the powerful feature extraction ability, existing Convolution Neural Network (CNN) and Transformers based NR-IQA methods have achieved considerable progress. However, they still exhibit limited capability when facing unknown authentic distortion datasets. To further improve NR-IQA performance, in this paper, a novel supervised contrastive learning (SCL) and Transformer-based NR-IQA model SaTQA is proposed. We first train a model on a large-scale synthetic dataset by SCL to extract degradation features of images with various distortion types and levels. To further extract distortion information, we propose a backbone network incorporating the Multi-Stream Block (MSB) by combining the CNN inductive bias and Transformer long-term dependence modeling capability. Finally, we propose the Patch Attention Block (PAB) to obtain the final distorted image quality score by fusing the degradation features learned from contrastive learning with the perceptual distortion information extracted by the backbone network. Experimental results on six standard IQA datasets show that SaTQA outperforms the state-of-the-art methods for both synthetic and authentic datasets.

Introduction

No-Reference Image Quality Assessment (NR-IQA) aims to evaluate image quality without a pristine reference. While deep learning has significantly advanced this field, a persistent challenge remains: the gap between synthetic distortions (e.g., Gaussian blur) and "authentic" real-world distortions (e.g., complex sensor noise or poor lighting). Existing methods often suffer from two main limitations:

  1. Feature Extraction Trade-offs: CNNs excel at local texture detection but lack global context, while pure Transformers capture global dependencies but may overlook fine-grained local details like edges.
  2. Generalization Issues: Many models fail to explicitly learn the distinction between various distortion types and levels, leading to poor performance on diverse, large-scale datasets.

Methodology

The proposed SaTQA framework addresses these gaps through a three-stage architecture that integrates contrastive pre-training with a hybrid backbone.

The SaTQA architecture featuring Supervised Contrastive Learning (SCL) for degradation feature extraction, Multi-Stream Blocks (MSB) for perceptual modeling, and Patch Attention Blocks (PAB) for score prediction.

Supervised Contrastive Learning (SCL)

Before quality scoring, the model undergoes a pre-training phase on a large synthetic dataset (KADIS). Using SCL, the model learns to cluster images with similar distortion types and levels in a feature space. This phase does not require human subjective scores, focusing purely on identifying the nature of the degradation.

Multi-Stream Block (MSB)

The MSB serves as the backbone, combining CNN inductive bias with Transformer global modeling. It processes features through three parallel streams: Deformable Convolutions to capture local edges, Depthwise Convolutions for efficient local feature processing, and Multi-Head Self-Attention (MHSA) to model long-range dependencies across the entire image.

Patch Attention Block (PAB)

The PAB uses a cross-attention mechanism to fuse the "degradation features" from the SCL phase with the "perceptual features" from the MSB backbone. This integration allows the model to leverage its prior knowledge of distortion types to inform the final quality score prediction.

Experimental Results

SaTQA was evaluated on several benchmarks, including synthetic (LIVE, CSIQ, TID2013) and authentic (LIVE Challenge, KONIQ, LIVE-FB) datasets.

Performance comparison of SaTQA against state-of-the-art methods (MANIQA, TReS, etc.) across multiple datasets, showing superior PLCC and SROCC metrics.

Quantitative Performance

SaTQA consistently outperformed state-of-the-art models across almost all benchmarks. Notably, on the challenging LIVE-FB dataset, SaTQA achieved a PLCC of 0.676, a significant jump over the previous best of 0.597. On the CSIQ dataset, it reached a near-perfect SROCC of 0.965. The model also demonstrated superior "zero-shot" performance when tested on datasets different from its training set, indicating a robust understanding of general quality principles.

Qualitative Insights

Grad-CAM visualizations indicate that SaTQA maintains a global perspective, considering the entire image structure rather than being distracted by specific semantic objects. This global awareness is crucial for identifying noise or artifacts that pervade the entire frame, whereas standard models often focus too narrowly on high-contrast semantic regions.

Conclusion

SaTQA represents a significant step forward in NR-IQA by effectively combining supervised contrastive learning with a hybrid CNN-Transformer architecture. By first learning to recognize the nature of image damage, the model builds a robust internal representation that generalizes well to unseen real-world images. Future work may extend this framework to video quality assessment or specialized domains like medical imaging and AI-generated content evaluation.

Summary updated

Abstract

Image Quality Assessment (IQA) has long been a research hotspot in the field of image processing, especially No-Reference Image Quality Assessment (NR-IQA). Due to the powerful feature extraction ability, existing Convolution Neural Network (CNN) and Transformers based NR-IQA methods have achieved considerable progress. However, they still exhibit limited capability when facing unknown authentic distortion datasets. To further improve NR-IQA performance, in this paper, a novel supervised contrastive learning (SCL) and Transformer-based NR-IQA model SaTQA is proposed. We first train a model on a large-scale synthetic dataset by SCL (no image subjective score is required) to extract degradation features of images with various distortion types and levels. To further extract distortion information from images, we propose a backbone network incorporating the Multi-Stream Block (MSB) by combining the CNN inductive bias and Transformer long-term dependence modeling capability. Finally, we propose the Patch Attention Block (PAB) to obtain the final distorted image quality score by fusing the degradation features learned from contrastive learning with the perceptual distortion information extracted by the backbone network. Experimental results on six standard IQA datasets show that SaTQA outperforms the state-of-the-art methods for both synthetic and authentic datasets. Code is available at https://github.com/I2-Multimedia-Lab/SaTQA.
BibTeX
@inproceedings{shi2024transformer,
  author = {Jinsong Shi and Pan Gao and Jie Qin},
  title = {{Transformer-Based No-Reference Image Quality Assessment via Supervised Contrastive Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=A9KsAfYnTK},
}
Figure