TL;DR
This paper introduces the LIVE-Meta Rendered Human Avatar VQA Database, a novel dataset for evaluating the quality of human avatar videos in VR. They also demonstrate its utility by benchmarking existing video quality assessment (VQA) models and proposing a new one, HoloQA, which shows promising results.
What Problem Does This Paper Address?
As virtual reality (VR) and augmented reality (AR) become more prevalent, the demand for high-quality, realistic human avatars in these environments is growing. Streaming these "digital twins" or holograms presents significant challenges, particularly concerning visual quality degradation due to compression and transmission artifacts. Existing video quality assessment (VQA) models, often designed for traditional 2D video, are insufficient for capturing the unique perceptual aspects of 3D human avatars in immersive VR settings. This gap hinders the development of efficient compression strategies and robust VQA tools for the metaverse.
Methods
The authors introduce the LIVE-Meta Rendered Human Avatar VQA Database, a comprehensive resource containing 720 human avatar videos. These videos were generated by applying 20 different distortion settings (including delays, color/depth resolution changes, and frame rate reductions) to 36 high-quality source avatar sequences. Crucially, these distorted videos were then rated by 78 human subjects using immersive 6-degrees-of-freedom (6DoF) VR headsets, providing a rich set of subjective quality judgments.
To process these subjective scores, they employed the SUREAL MLE-MOS method, a robust statistical approach that accounts for individual subject biases and inconsistencies, providing more reliable Mean Opinion Scores (MOS).
They also evaluated a range of state-of-the-art Full Reference (FR) and No Reference (NR) VQA models on their new dataset. Additionally, they propose and evaluate HoloQA, a novel VQA model designed specifically for human avatars, which utilizes a Mixture-of-Experts approach to capture both low-level pixel quality and high-level content-aware features.
Datasets
The primary contribution is the LIVE-Meta Rendered Human Avatar VQA Database. This dataset comprises:
- 36 source human avatar videos from Metastage, featuring diverse demographics and poses.
- 720 distorted videos created by applying 20 different encoding parameter combinations.
- Subjective quality ratings from 78 participants using Oculus Quest Pro VR headsets.
This dataset is notable for its focus on human avatars in VR, its use of 6DoF VR for data collection, and its comprehensive set of distortions relevant to streaming scenarios.
Key Insights and Ideas
- VR Headsets are Crucial: Subjective studies using VR headsets are essential for accurately capturing the perceptual quality of immersive 3D content, as they account for factors like immersion and 6DoF movement that 2D displays cannot replicate.
- Temporal Distortions Matter Most: The study found that temporal distortions (like frame rate reduction) had a more significant impact on perceived avatar quality than delays. This suggests that optimizing frame rates might be a more effective strategy for bandwidth savings than solely focusing on latency.
- Content Matters, but Less Than Expected: While different avatar contents showed some variation in ambiguity, the impact of distortions like color resolution and frame rate was more pronounced than content-specific differences.
- Existing VQA Models Struggle: Many standard VQA models, even those adapted for 3D, showed limited performance on this specialized dataset, highlighting the need for avatar-specific VQA.
- HoloQA Shows Promise: The proposed HoloQA model demonstrated competitive performance, suggesting that specialized models can better capture the nuances of human avatar quality.
Key Results
- The LIVE-Meta Rendered Human Avatar VQA Database is the largest 3D graphics VQA database to date, featuring 720 videos and 78 human subjects.
- Subjective studies revealed that color resolution and frame rate distortions significantly impact perceived quality, more so than delay artifacts.
- FR and NR VQA models were evaluated, with CONTRIQUE (FR), Re-IQA (FR), CONVIQT (FR), and HoloQA showing the best performance among the tested models.
- The proposed HoloQA model achieved state-of-the-art performance on the LIVE-Meta Rendered Human Avatar VQA database.
Limitations and Future Work
- VR Headset Specificity: The study used Oculus Quest Pro headsets, which might limit the generalizability of findings to other VR environments and devices. Future work could explore a wider range of VR hardware.
- Distortion Types: While the dataset covers key distortions, future studies could incorporate additional distortion types like color errors, edge noise, and surface noise to further enrich the VQA evaluation.
- Model Development: The authors suggest future research should focus on developing more advanced deep learning methodologies specifically for FR and NR IQA/VQA algorithms tailored for avatar analysis.
- Broader VR Environments: Exploring diverse VR environments and headsets is recommended to gain a more comprehensive understanding of device factors influencing VR video quality.
Summary updated