TL;DR
This paper introduces the first large-scale database for assessing the visual quality of 3D digital human heads (DHHs) and proposes a novel full-reference quality assessment method. Their approach leverages a Swin Transformer for feature extraction and multi-head attention for fusion, achieving state-of-the-art performance in predicting subjective quality ratings.
What Problem Does This Paper Address?
The rapid advancement in creating realistic digital humans for various applications (gaming, metaverse, etc.) has outpaced the development of reliable methods to objectively assess their visual quality. While generating these digital assets is complex and expensive, evaluating their perceptual fidelity has been a significant challenge. Existing 3D quality assessment (3D-QA) databases and methods often focus on general 3D models like point clouds or meshes, but lack specificity for the nuances of digital human heads, which have unique perceptual characteristics. This gap hinders the development and deployment of high-fidelity digital human technologies.
Methods
The authors propose a full-reference (FR) projection-based quality assessment method. This means their method compares a distorted digital human head to its original, pristine version. The core of their approach involves:
- Hierarchical Feature Extraction: They utilize a pre-trained Swin Transformer tiny model. This network is chosen for its efficiency and ability to capture features at different levels of abstraction, from low-level details to high-level semantic information, mimicking human visual perception. Features are extracted from multiple stages of the Swin Transformer.
- Multi-Head Attention for Feature Fusion: To effectively combine the quality-aware information from the reference and distorted digital human head representations, they employ a multi-head attention mechanism. This allows the model to learn how to weigh and integrate different features to better understand their relationship and impact on perceived quality.
- Feature Regression: The fused features are then passed through fully-connected layers to predict a final quality score.
- Loss Function: An L1 loss is used during training to minimize the difference between predicted and ground truth quality scores.
Crucially, their method operates on rendered projections (specifically, front-view images) of the 3D models, allowing them to leverage mature image quality assessment (IQA) techniques and models.
Datasets
The paper introduces the Digital Human Head Quality Assessment (DHHQA) database, which is presented as the first large-scale benchmark specifically for DHHs.
- Source Models: 55 high-quality, textured 3D digital human head models were selected, representing diverse demographics (male/female, young/old).
- Distorted Models: These reference models were degraded using 7 types of common distortions (surface simplification, position compression, UV map compression, texture down-sampling, texture compression, geometry noise, and color noise) at 4 different levels each. This resulted in a total of 1,540 distorted DHH models.
- Subjective Ratings: A well-controlled subjective experiment was conducted with 255 participants to collect Mean Opinion Scores (MOS) for the distorted DHHs.
The DHHQA database is made publicly available.
Key Insights and Ideas
- The Need for Specialized DHH Quality Assessment: The paper strongly argues that general 3D QA methods are insufficient for digital human heads, necessitating a dedicated database and assessment approach.
- Projection-Based Methods Leverage Existing IQA Success: By rendering 3D models into 2D projections, the authors effectively bridge the gap between 3D QA and the well-established field of image quality assessment, enabling the use of powerful deep learning models like Swin Transformers.
- Hierarchical Features are Crucial: The success of the Swin Transformer highlights the importance of capturing visual information at multiple scales and levels of abstraction for accurate perceptual quality prediction.
- Attention Mechanisms Enhance Feature Integration: Multi-head attention proves effective in fusing information from reference and distorted inputs, allowing the model to focus on perceptually relevant differences.
Key Results
- The proposed DHHQA database is the largest of its kind for digital human heads.
- The proposed FR projection-based quality assessment method achieved state-of-the-art performance, outperforming mainstream FR-IQA metrics.
- Specifically, the proposed method achieved the highest SRCC (0.9286), PLCC (0.9320), and KRCC (0.7585) scores on their DHHQA database, with a significantly lower RMSE (7.2910) compared to other methods. For instance, it was approximately 0.03 higher in SRCC than the next best competitor, LPIPS.
Limitations and Future Work
- Limitations: The proposed method is a full-reference method, meaning it requires the original, undistorted digital human head model for comparison. This can be a limitation in scenarios where only the distorted version is available.
- Future Work: The authors suggest that their work lays the foundation for developing no-reference (NR) quality assessment methods for digital human heads, which would be more practical in many real-world applications. Further exploration into different distortion types and more complex digital human models could also be beneficial.
Summary updated