No-Reference Image Quality Assessment Based on Multi-Task Generative Adversarial Network

Yao Ma, Xibiao Cai, Fuming Sun, Shijie HaoIEEE Access 2019 · DOI

Abstract

Since human observers are the ultimate receivers of an image, most image quality assessment (IQA) methods are based on analysis of the properties and mechanisms of the human visual system. However, due to the lack of undistorted images for reference, the accuracy of no-reference IQA (NR-IQA) often cannot compete with that of full-reference IQA (FR-IQA). To bridge this performance gap, we propose a NR-IQA method based on a multi-task generative adversarial network (GAN), which attempts to restore dependable hallucinated images to compensate for missing reference images. Two tasks—hallucination images and quality maps—are outputted by the generator and combined with a specific loss to improve reliability. Additionally, two discriminator networks are used to distinguish undistorted images from hallucinated pairs and quality maps from structural similarity index measurement (SSIM) maps. Finally, the hallucinated and distorted images are input into an IQA network to evaluate quality scores. The superiority of our proposed method is verified by experiments on the LIVE, TID2008, and TID2013 datasets.

Introduction

Image Quality Assessment (IQA) is essential for a wide range of digital imaging applications, from social media filters to medical diagnostics. Traditionally, IQA is categorized into Full-Reference (FR-IQA), which compares a distorted image to a perfect original, and No-Reference (NR-IQA), which evaluates an image without any ground truth.

While FR-IQA is highly accurate due to its comparative nature, NR-IQA is far more practical for real-world scenarios where the original image is unavailable. However, NR-IQA typically lags in performance because models struggle to "know" what a perfect version of a specific distorted image should look like. This research aims to bridge that gap by using deep learning to "hallucinate" the missing reference information.

Methodology

The core of the proposed method is a Multi-Task Generative Adversarial Network. Instead of directly regressing a quality score, the system first attempts to reconstruct the missing reference data to enable a comparative evaluation similar to FR-IQA.

High-level flowchart of the proposed NR-IQA method.

The Multi-Task GAN Architecture

The system operates in two main stages, utilizing a complex generator and dual-discriminator setup:

  1. The Generator (U-Net Architecture): The generator takes a distorted image and performs two tasks simultaneously:
  2. Hallucination: Reconstructing a clean, undistorted version of the image.
  3. Quality Mapping: Generating a predicted SSIM map, which acts as an auxiliary guide to help the network understand the spatial distribution of degradation.
  4. Dual Discriminators: To ensure high fidelity, the authors use two separate discriminator networks ($D_r$ and $D_s$). $D_r$ checks if the hallucinated image looks "real" compared to actual undistorted images, while $D_s$ ensures the generated quality map matches the statistical properties of real SSIM maps.
  5. IQA-Specific Loss: Standard GANs often struggle with fine textures. To address this, the authors introduced a specific "IQA-loss" based on features from a pre-trained regression network, ensuring the hallucinated image is useful for quality measurement rather than just visually pleasing.
Detailed architecture of the multi-task GAN, including the generator and dual discriminators.

The IQA Network

In the final stage, the distorted image and the hallucinated reference are fed into an IQA network. This network extracts features from both images, fuses them, and regresses a final quality score based on the differences between the two.

Experimental Results

The model was trained and evaluated on three gold-standard IQA datasets: LIVE (779 images), TID2008 (1,700 images), and TID2013 (3,000 images).

Key Insights

  • Hallucination as a Bridge: Treating NR-IQA as a "missing data" problem allows the model to leverage the powerful comparative logic of FR-IQA.
  • Multi-Tasking for Stability: Forcing the network to predict a quality map (SSIM) alongside the hallucinated image provides a "predetermined direction," making the learning process more robust and stable.
  • Performance: The proposed method demonstrated superior performance compared to several state-of-the-art NR-IQA algorithms like HOSA and RankIQA. On the TID2013 dataset, the model achieved a Spearman Rank Correlation Coefficient (SROCC) of 0.914.

Conclusion and Future Work

While the multi-task GAN approach significantly improves NR-IQA accuracy, some limitations remain. In cases of extreme distortion where texture details are completely lost, the GAN may struggle to hallucinate accurate references, leading to a drop in prediction accuracy. Additionally, the computational complexity of GAN ensembles may pose challenges for real-time mobile deployment. Future research will focus on improving the model's generalization to "wild" images containing multiple, overlapping distortion types.

Summary updated

Abstract

Since human observers are the ultimate receivers of an image, most of the image quality assessment (IQA) methods are based on analysis of the properties and mechanism of the human visual system. However, due to the lack of undistorted images for a reference, the accuracy of the no-reference IQA (NR-IQA) cannot compete with that of the full-reference IQA (FR-IQA). To bridge the performance gap between the FR-IQA and NR-IQA methods, we propose a NR-IQA method based on multi-task generative adversarial network, which attempts to restore dependable hallucinated images to compensate for the missing corresponding reference images. Two tasks, hallucination images and the quality maps are outputted by the generator and are combined with the specific loss to improve the reliability of hallucination images. Besides, two discriminator networks are used to respectively distinguish the undistorted images and hallucination images pairs, quality maps and structural similarity index measurement maps pairs. Finally, the hallucination images and distorted images are input into the IQA network, and quality scores are evaluated based on the differences between them. The superiority of our proposed method is verified by several different experiments on the LIVE datasets, TID2008 datasets, and TID2013 datasets.
BibTeX
@inproceedings{ma2019no,
  author = {Yao Ma and Xibiao Cai and Fuming Sun and Shijie Hao},
  title = {{No-Reference Image Quality Assessment Based on Multi-Task Generative Adversarial Network}},
  booktitle = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2942625},
}
Figure