Spatial Attention-based Non-reference Perceptual Quality Prediction Network for Omnidirectional Images

Li Yang, Mai Xu, Xin Deng, Bo Feng

Abstract

Predicting the quality of 360-degree omnidirectional images (ODIs) is a critical task for Virtual Reality (VR) applications, yet it is complicated by high resolutions, projection-induced distortions, and the frequent absence of high-quality reference images. This paper proposes SAP-net, a novel no-reference (NR) quality assessment tool that leverages self-attention to identify perceptually significant regions without the need for costly human eye-tracking data. By utilizing a Wavelet-based Residual Enhancement module to create a pseudo-reference and Residual Spatial Attention Blocks for quality estimation, SAP-net achieves state-of-the-art performance. Experimental results on the newly introduced IQA-ODI dataset demonstrate that SAP-net significantly outperforms existing NR methods and even surpasses several full-reference metrics in consistency with human subjective scores.

Introduction

Omnidirectional images (ODIs) are foundational to immersive VR experiences but present unique challenges for Image Quality Assessment (IQA). These images are typically massive (8K or higher) and must be projected from a spherical format onto a 2D plane, introducing geometric distortions. Furthermore, heavy compression is often required for streaming, which further degrades quality. Traditional metrics like PSNR or SSIM require a "perfect" reference image, which is rarely available in live streaming or user-generated content scenarios. Existing NR methods often rely on explicit human saliency maps—data indicating where viewers look—which is expensive and time-consuming to collect. SAP-net addresses these issues by implicitly learning human attention through a self-attention mechanism, reducing computational overhead and eliminating the need for saliency labels.

Methods

The SAP-net architecture consists of three primary components designed to mimic the human visual system's response to distortions:

  1. Wavelet-based Residual Enhancement (WBRE): Standard pooling layers often discard high-frequency details essential for quality assessment. WBRE uses the Discrete Wavelet Transform (DWT) to attempt a reconstruction of a high-quality version of the impaired image, serving as a "pseudo-reference."
  2. Perceptual Quality Estimation (PQE): This module compares the impaired image with the pseudo-reference to generate an error map. It employs Residual Spatial Attention Blocks (RSAB) to perform self-attention, allowing the network to focus on regions most likely to impact human perception without requiring external eye-tracking data.
  3. Quality Regression (QR): The final stage integrates the perceptual quality map and error map to output a quality score consistent with human perception.

The IQA-ODI Dataset

To train and validate the model, the authors introduced the IQA-ODI benchmark. This dataset contains 1,080 images, including 120 high-quality 8K references and 960 impaired versions. Impairments include four levels of JPEG compression applied across four different projection patterns: Cube Map (CMP), Icosahedral (ISP), Octahedral (OHP), and Craster Parabolic (CPP). Subjective ground truth was established by 200 participants using HTC Vive VR headsets, providing Differential Mean Opinion Scores (DMOS).

Example of an omnidirectional image in equirectangular projection with a zoomed-in view highlighting compression artifacts.

Results and Discussion

SAP-net was benchmarked against nine state-of-the-art methods. It achieved a Pearson Linear Correlation Coefficient (PLCC) of 0.9258 and a Spearman Rank-Order Correlation Coefficient (SROCC) of 0.9036, outperforming the next best NR method, dipIQ (PLCC 0.8741).

A key insight from the IQA-ODI dataset analysis is that JPEG compression levels have a far more substantial impact on perceived quality than the specific geometric projection used. As shown in Figure 4, the distribution of DMOS remains relatively consistent across different projection types, suggesting that the artifacts introduced by compression are the primary drivers of quality degradation in VR environments.

Box plot showing the distribution of DMOS across different projection types (CPP, CMP, ISP, OHP), illustrating that projection type has a secondary effect compared to compression.

Conclusion

SAP-net provides a highly efficient and accurate solution for no-reference 360-degree image quality assessment. By replacing standard pooling with DWT and utilizing implicit spatial attention, it avoids the need for expensive saliency data and complex viewport extraction. Future research will focus on extending this spatial attention mechanism to the temporal domain for 360-degree video and expanding the dataset to include a wider variety of real-world distortions such as sensor noise and transmission loss.

Summary updated

Abstract

Due to the strong correlation between visual attention and perceptual quality, many methods attempt to use human saliency information for image quality assessment. Although this mechanism can get good performance, the networks require human saliency labels, which is not easily accessible for omnidirectional images (ODI). To alleviate this issue, we propose a spatial attention-based perceptual quality prediction network for non-reference quality assessment on ODIs (SAP-net). To drive our SAP-net, we establish a large-scale IQA dataset of ODIs (IQA-ODI), which is composed of subjective scores of 200 subjects on 1,080 ODIs. In IQA-ODI, there are 120 high quality ODIs as reference, and 960 ODIs with impairments in both JPEG compression and map projection. Without any human saliency labels, our network can adaptively estimate human perceptual quality on impaired ODIs through a self-attention manner, which significantly promotes the prediction performance of quality scores. Moreover, our method greatly reduces the computational complexity in quality assessment task on ODIs. Extensive experiments validate that our network outperforms 9 state-of-the-art methods for quality assessment on ODIs. The dataset and code have been available on \url{ https://github.com/yanglixiaoshen/SAP-Net}.
BibTeX
@inproceedings{yang2021spatial,
  author = {Li Yang and Mai Xu and Xin Deng and Bo Feng},
  title = {{Spatial Attention-based Non-reference Perceptual Quality Prediction Network for Omnidirectional Images}},
  booktitle = {ICME},
  year = {2021},
  url = {https://openreview.net/forum?id=ilozPCija7zc},
}
Figure