Perceptual Quality Assessment of Omnidirectional Images as Moving Camera Videos

Xiangjie Sui, Kede Ma, Yiru Yao, Yuming FangIEEE TVCG 2022 · DOI

TL;DR

This paper tackles the challenge of assessing the perceived quality of 360° images in virtual reality (VR) by proposing a novel framework that treats them as moving camera videos. By incorporating user viewing conditions (starting point and exploration time) and behaviors (scanpaths), their method significantly improves objective quality assessment, outperforming existing approaches.

What Problem Does This Paper Address?

Traditional image quality assessment (IQA) methods, designed for 2D images, struggle when applied to omnidirectional (360°) images used in VR. This is because VR viewing experiences are fundamentally different: users can freely explore the scene, and their perception of quality is heavily influenced by how they explore it. Factors like where they start looking (starting point) and how long they spend exploring (exploration time) dictate their "scanpath" – the trajectory of their gaze. Existing objective IQA models often ignore these crucial viewing conditions and behaviors, leading to inaccurate quality predictions. This paper aims to bridge this gap by developing an objective quality assessment framework that accounts for these user-centric factors.

Methods

The core innovation of this paper lies in its "omnidirectional image-to-video conversion" approach. Instead of directly assessing the static 360° image, they transform it into a sequence of video frames. This is achieved by:

  1. Simulating User Exploration: Using the defined viewing conditions (starting point, exploration time) and a user's scanpath, they extract a series of viewports (rectilinear projections) from the omnidirectional image. This creates a video sequence that mimics how a user would naturally explore the scene with a moving camera.
  2. Leveraging 2D Video Quality Assessment (VQA): Once the 360° image is converted into a video sequence, existing, advanced 2D VQA models can be applied to assess its quality.
  3. Temporal Pooling: Frame-level quality scores from the VQA models are then combined using temporal pooling strategies (like a temporal hysteresis model) to derive a final, overall quality score for the omnidirectional image.

This approach allows them to adapt well-established 2D IQA/VQA techniques to the unique challenges of omnidirectional content.

Datasets

The authors introduce a new dataset for evaluating their method, which includes:

  • Proposed Database: Consists of 72 images (36 reference, 36 distorted) with two types of distortions: stitching distortion and H.265 compression. This database was specifically designed to study the impact of viewing conditions and behaviors.
  • OIQA Database [10]: A publicly available database with 320 distorted omnidirectional images and four distortion types (JPEG, JPEG2000, Gaussian noise, Gaussian blur).
  • LIVE 3D VR IQA Database [13]: Another publicly available dataset containing 450 distorted stereoscopic omnidirectional images with six distortion types.

Key Insights and Ideas

  • Viewing Conditions Matter: The study strongly validates that the starting point and exploration time significantly influence perceived quality, especially for localized distortions like stitching artifacts.
  • Distortion Type is Key: The impact of viewing conditions is highly dependent on the type of distortion. Localized distortions are more sensitive to viewing behavior than global, uniform distortions like compression.
  • Video Conversion is a Powerful Abstraction: Treating 360° images as moving camera videos by simulating user exploration provides a robust way to incorporate viewing behaviors and leverage existing 2D VQA tools.
  • 2D IQA Advances Transfer Well: Surprisingly, recent 2D IQA models applied directly to equirectangular projections often outperformed older, specialized omnidirectional IQA methods, suggesting the field of 2D IQA has advanced significantly.

Key Results

  • The proposed framework, particularly with advanced 2D IQA models like O-NLPD and O-DISTS, achieved state-of-the-art performance on the proposed database and competitive results on the OIQA and LIVE 3D VR IQA databases.
  • Their models demonstrated significant performance improvements over projection-based and viewport-based methods, especially in handling localized distortions.
  • Statistical significance tests confirmed that their proposed models perform significantly better than competing methods.

Limitations and Future Work

  • Localized Distortion Handling: While the framework shows promise, handling localized distortions like stitching artifacts remains a challenge for existing objective quality models, even when incorporated into their framework. Future work could focus on developing IQA models specifically adept at quantifying these types of distortions.
  • Display Constraints: The current work assumes fixed display constraints. Future research could incorporate display information (e.g., Field of View, resolution) as additional viewing conditions.
  • Personalized IQA: The framework can be extended to personalized IQA by leveraging individual user behavior statistics, rather than averaging across users.
  • VR Discomfort: Predicting overall quality-of-experience, including VR discomfort and sickness, is a promising future direction.
  • Scanpath Pattern Impact: While tested, the impact of different scanpath patterns on prediction accuracy was not found to be significant on globally distorted images, suggesting further investigation with more diverse distortions might be needed.

Summary updated

BibTeX
@article{sui2022perceptual,
  author = {Xiangjie Sui and Kede Ma and Yiru Yao and Yuming Fang},
  title = {{Perceptual Quality Assessment of Omnidirectional Images as Moving Camera Videos}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2022},
  doi = {10.1109/tvcg.2021.3050888},
}

Loading related content...

Figure