Blind Omnidirectional Image Quality Assessment: Integrating Local Statistics and Global Semantics

Wei Zhou, Zhou Wang

TL;DR

Researchers from the University of Waterloo have developed $S^2$, a new "blind" (no-reference) method for assessing the quality of 360-degree omnidirectional images. By mimicking how humans browse VR environments—combining local visual details from specific viewports with a high-level "global" understanding of the scene—the model achieves state-of-the-art performance in predicting how humans perceive image quality.

What Problem Does This Paper Address?

Omnidirectional Image Quality Assessment (OIQA) is challenging because, unlike standard 2D photos, a viewer never sees the entire 360-degree field of view at once. Instead, they browse through various "viewports."

Existing quality metrics often struggle with two things:

  1. Reference Dependency: Many models require a "perfect" original image to compare against (Full-Reference), which isn't always available in real-world streaming or VR applications.
  2. The Semantic Gap: Most models focus either on low-level pixel statistics (like blur or noise) or high-level content (like "is this a classroom?"), but rarely both. Humans, however, integrate these cues; we notice if a "classroom" looks "grainy."

Methods

The authors propose the $S^2$ framework, which uses two parallel paths to evaluate an image:

  • The Statistic Path (Local): The model samples $N$ local viewports from the 360-degree image. It then builds multi-scale Gaussian and Laplacian pyramids for these viewports. From these, it extracts:
  • Local Binary Patterns (LBP): To capture texture and structural statistics.
  • Natural Scene Statistics (NSS): To identify deviations from the "natural" look of an image caused by distortions.
  • The Semantic Path (Global): The model looks at the "hallucinated" global omnidirectional image (the projected map). It uses a pre-trained VGG-M network (a deep convolutional neural network) to extract high-level semantic features.
  • Fusion: Features from both paths are fed into separate Support Vector Regression (SVR) models to predict quality scores, which are then combined via a weighted average to produce the final quality score.

Datasets

The model was evaluated on the CVIQD (Compressed VR Image Quality Database).

  • Composition: 16 original images and 528 distorted versions.
  • Distortions: Images were compressed using three common standards: JPEG, AVC (H.264), and HEVC (H.265).
  • Ground Truth: The dataset includes Mean Opinion Scores (MOS) from human subjects, providing a benchmark for "perceptual" quality.

Key Insights and Ideas

  • Semantic Confidence vs. Distortion: One of the most interesting findings is that as image distortion increases, the "confidence" of the semantic model (its ability to recognize the scene) decreases. Essentially, heavy noise or blur makes it harder for the AI to "understand" what it's looking at, mirroring human frustration with low-quality content.
  • HEVC is "Smarter": The researchers found that semantic confidence drops much more slowly in images compressed with HEVC compared to older JPEG or AVC encoders, suggesting HEVC preserves meaningful scene structures better at lower bitrates.
  • The "All" Case Wins: Ablation studies showed that while the Statistic Path is the more dominant factor in predicting quality, the Semantic Path provides "complementary information" that pushes the model's accuracy to the top of the leaderboard.
  • Efficiency: The model is surprisingly robust to the number of viewports sampled. Using just 6 viewports yielded performance nearly identical to using 80, which is a massive win for computational efficiency.

Key Results

The $S^2$ method outperformed all other tested No-Reference (NR) and even several Full-Reference (FR) models. On the CVIQD dataset, it achieved:

  • SROCC (Spearman Correlation): 0.9710
  • PLCC (Pearson Correlation): 0.9781
  • RMSE (Root Mean Squared Error): 2.8945

For context, it beat the previous state-of-the-art NR-OIQA model (MFILGN) which had an SROCC of 0.9670.

Limitations and Future Work

While the model is highly effective, the authors note a few areas for growth:

  • Backbone Sensitivity: The performance varies slightly depending on which deep learning architecture (VGG-F, VGG-M, VGG-S) is used for the semantic path.
  • Future Directions: The authors suggest that even more advanced models for statistic and semantic analysis could be integrated into the $S^2$ framework in the future to further refine Quality of Experience (QoE) assessments for immersive VR systems.
  • Static vs. Dynamic: This paper focuses on static images; extending this dual-path approach to 360-degree video (where temporal statistics matter) would be a logical next step.

Summary updated

Abstract

Omnidirectional image quality assessment (OIQA) aims to predict the perceptual quality of omnidirectional images that cover the whole 180$\times$360$^{\circ}$ viewing range of the visual environment. Here we propose a blind/no-reference OIQA method named S$^2$ that bridges the gap between low-level statistics and high-level semantics of omnidirectional images. Specifically, statistic and semantic features are extracted in separate paths from multiple local viewports and the hallucinated global omnidirectional image, respectively. A quality regression along with a weighting process is then followed that maps the extracted quality-aware features to a perceptual quality prediction. Experimental results demonstrate that the proposed S$^2$ method offers highly competitive performance against state-of-the-art methods.
BibTeX
@inproceedings{zhou2023blind,
  author = {Wei Zhou and Zhou Wang},
  title = {{Blind Omnidirectional Image Quality Assessment: Integrating Local Statistics and Global Semantics}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222049},
}

Loading related content...

Figure