Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual Reality

Yu-Chih Chen, Avinab Saha, Alexandre Chapiro, Christian Häne, Jean-Charles Bazin, Bo Qiu, Stefano Zanetti, Ioannis Katsavounidis, Alan C. BovikIEEE TIP 2024 · DOI · OpenReview

TL;DR

This paper introduces the LIVE-Meta Rendered Human Avatar VQA Database, a novel dataset for evaluating the quality of human avatar videos in VR. They also demonstrate its utility by benchmarking existing video quality assessment (VQA) models and proposing a new one, HoloQA, which shows promising results.

What Problem Does This Paper Address?

As virtual reality (VR) and augmented reality (AR) become more prevalent, the demand for high-quality, realistic human avatars in these environments is growing. Streaming these "digital twins" or holograms presents significant challenges, particularly concerning visual quality degradation due to compression and transmission artifacts. Existing video quality assessment (VQA) models, often designed for traditional 2D video, are insufficient for capturing the unique perceptual aspects of 3D human avatars in immersive VR settings. This gap hinders the development of efficient compression strategies and robust VQA tools for the metaverse.

Methods

The authors introduce the LIVE-Meta Rendered Human Avatar VQA Database, a comprehensive resource containing 720 human avatar videos. These videos were generated by applying 20 different distortion settings (including delays, color/depth resolution changes, and frame rate reductions) to 36 high-quality source avatar sequences. Crucially, these distorted videos were then rated by 78 human subjects using immersive 6-degrees-of-freedom (6DoF) VR headsets, providing a rich set of subjective quality judgments.

To process these subjective scores, they employed the SUREAL MLE-MOS method, a robust statistical approach that accounts for individual subject biases and inconsistencies, providing more reliable Mean Opinion Scores (MOS).

They also evaluated a range of state-of-the-art Full Reference (FR) and No Reference (NR) VQA models on their new dataset. Additionally, they propose and evaluate HoloQA, a novel VQA model designed specifically for human avatars, which utilizes a Mixture-of-Experts approach to capture both low-level pixel quality and high-level content-aware features.

Datasets

The primary contribution is the LIVE-Meta Rendered Human Avatar VQA Database. This dataset comprises:

  • 36 source human avatar videos from Metastage, featuring diverse demographics and poses.
  • 720 distorted videos created by applying 20 different encoding parameter combinations.
  • Subjective quality ratings from 78 participants using Oculus Quest Pro VR headsets.

This dataset is notable for its focus on human avatars in VR, its use of 6DoF VR for data collection, and its comprehensive set of distortions relevant to streaming scenarios.

Key Insights and Ideas

  • VR Headsets are Crucial: Subjective studies using VR headsets are essential for accurately capturing the perceptual quality of immersive 3D content, as they account for factors like immersion and 6DoF movement that 2D displays cannot replicate.
  • Temporal Distortions Matter Most: The study found that temporal distortions (like frame rate reduction) had a more significant impact on perceived avatar quality than delays. This suggests that optimizing frame rates might be a more effective strategy for bandwidth savings than solely focusing on latency.
  • Content Matters, but Less Than Expected: While different avatar contents showed some variation in ambiguity, the impact of distortions like color resolution and frame rate was more pronounced than content-specific differences.
  • Existing VQA Models Struggle: Many standard VQA models, even those adapted for 3D, showed limited performance on this specialized dataset, highlighting the need for avatar-specific VQA.
  • HoloQA Shows Promise: The proposed HoloQA model demonstrated competitive performance, suggesting that specialized models can better capture the nuances of human avatar quality.

Key Results

  • The LIVE-Meta Rendered Human Avatar VQA Database is the largest 3D graphics VQA database to date, featuring 720 videos and 78 human subjects.
  • Subjective studies revealed that color resolution and frame rate distortions significantly impact perceived quality, more so than delay artifacts.
  • FR and NR VQA models were evaluated, with CONTRIQUE (FR), Re-IQA (FR), CONVIQT (FR), and HoloQA showing the best performance among the tested models.
  • The proposed HoloQA model achieved state-of-the-art performance on the LIVE-Meta Rendered Human Avatar VQA database.

Limitations and Future Work

  • VR Headset Specificity: The study used Oculus Quest Pro headsets, which might limit the generalizability of findings to other VR environments and devices. Future work could explore a wider range of VR hardware.
  • Distortion Types: While the dataset covers key distortions, future studies could incorporate additional distortion types like color errors, edge noise, and surface noise to further enrich the VQA evaluation.
  • Model Development: The authors suggest future research should focus on developing more advanced deep learning methodologies specifically for FR and NR IQA/VQA algorithms tailored for avatar analysis.
  • Broader VR Environments: Exploring diverse VR environments and headsets is recommended to gain a more comprehensive understanding of device factors influencing VR video quality.

Summary updated

Abstract

We study the visual quality judgments of human subjects on digital human avatars (sometimes referred to as “holograms” in the parlance of virtual reality [VR] and augmented reality [AR] systems) that have been subjected to distortions. We also study the ability of video quality models to predict human judgments. As streaming human avatar videos in VR or AR become increasingly common, the need for more advanced human avatar video compression protocols will be required to address the tradeoffs between faithfully transmitting high-quality visual representations while adjusting to changeable bandwidth scenarios. During transmission over the internet, the perceived quality of compressed human avatar videos can be severely impaired by visual artifacts. To optimize trade-offs between perceptual quality and data volume in practical workflows, video quality assessment (VQA) models are essential tools. However, there are very few VQA algorithms developed specifically to analyze human body avatar videos, due, at least in part, to the dearth of appropriate and comprehensive datasets of adequate size. Towards filling this gap, we introduce the LIVE-Meta Rendered Human Avatar VQA Database, which contains 720 human avatar videos processed using 20 different combinations of encoding parameters, labeled by corresponding human perceptual quality judgments that were collected in six degrees of freedom VR headsets. To demonstrate the usefulness of this new and unique video resource, we use it to study and compare the performances of a variety of state-of-the-art Full Reference and No Reference video quality prediction models, including a new model called HoloQA. As a service to the research community, we publicly releases the metadata of the new database at https://live.ece.utexas.edu/research/LIVE-Meta-rendered-human-avatar/index.html .
BibTeX
@article{chen2024subjective,
  author = {Yu-Chih Chen and Avinab Saha and Alexandre Chapiro and Christian Häne and Jean-Charles Bazin and Bo Qiu and Stefano Zanetti and Ioannis Katsavounidis and Alan C. Bovik},
  title = {{Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual Reality}},
  journal = {IEEE Trans. Image Process.},
  year = {2024},
  doi = {10.48550/arxiv.2408.07041},
}

Loading related content...

Figure