Perceptual Quality Assessment for Digital Human Heads

Zicheng Zhang, Yingjie Zhou, Wei Sun, Xiongkuo Min, Guangtao Zhai

TL;DR

This paper introduces the first large-scale database for assessing the visual quality of 3D digital human heads (DHHs) and proposes a novel full-reference quality assessment method. Their approach leverages a Swin Transformer for feature extraction and multi-head attention for fusion, achieving state-of-the-art performance in predicting subjective quality ratings.

What Problem Does This Paper Address?

The rapid advancement in creating realistic digital humans for various applications (gaming, metaverse, etc.) has outpaced the development of reliable methods to objectively assess their visual quality. While generating these digital assets is complex and expensive, evaluating their perceptual fidelity has been a significant challenge. Existing 3D quality assessment (3D-QA) databases and methods often focus on general 3D models like point clouds or meshes, but lack specificity for the nuances of digital human heads, which have unique perceptual characteristics. This gap hinders the development and deployment of high-fidelity digital human technologies.

Methods

The authors propose a full-reference (FR) projection-based quality assessment method. This means their method compares a distorted digital human head to its original, pristine version. The core of their approach involves:

  • Hierarchical Feature Extraction: They utilize a pre-trained Swin Transformer tiny model. This network is chosen for its efficiency and ability to capture features at different levels of abstraction, from low-level details to high-level semantic information, mimicking human visual perception. Features are extracted from multiple stages of the Swin Transformer.
  • Multi-Head Attention for Feature Fusion: To effectively combine the quality-aware information from the reference and distorted digital human head representations, they employ a multi-head attention mechanism. This allows the model to learn how to weigh and integrate different features to better understand their relationship and impact on perceived quality.
  • Feature Regression: The fused features are then passed through fully-connected layers to predict a final quality score.
  • Loss Function: An L1 loss is used during training to minimize the difference between predicted and ground truth quality scores.

Crucially, their method operates on rendered projections (specifically, front-view images) of the 3D models, allowing them to leverage mature image quality assessment (IQA) techniques and models.

Datasets

The paper introduces the Digital Human Head Quality Assessment (DHHQA) database, which is presented as the first large-scale benchmark specifically for DHHs.

  • Source Models: 55 high-quality, textured 3D digital human head models were selected, representing diverse demographics (male/female, young/old).
  • Distorted Models: These reference models were degraded using 7 types of common distortions (surface simplification, position compression, UV map compression, texture down-sampling, texture compression, geometry noise, and color noise) at 4 different levels each. This resulted in a total of 1,540 distorted DHH models.
  • Subjective Ratings: A well-controlled subjective experiment was conducted with 255 participants to collect Mean Opinion Scores (MOS) for the distorted DHHs.

The DHHQA database is made publicly available.

Key Insights and Ideas

  • The Need for Specialized DHH Quality Assessment: The paper strongly argues that general 3D QA methods are insufficient for digital human heads, necessitating a dedicated database and assessment approach.
  • Projection-Based Methods Leverage Existing IQA Success: By rendering 3D models into 2D projections, the authors effectively bridge the gap between 3D QA and the well-established field of image quality assessment, enabling the use of powerful deep learning models like Swin Transformers.
  • Hierarchical Features are Crucial: The success of the Swin Transformer highlights the importance of capturing visual information at multiple scales and levels of abstraction for accurate perceptual quality prediction.
  • Attention Mechanisms Enhance Feature Integration: Multi-head attention proves effective in fusing information from reference and distorted inputs, allowing the model to focus on perceptually relevant differences.

Key Results

  • The proposed DHHQA database is the largest of its kind for digital human heads.
  • The proposed FR projection-based quality assessment method achieved state-of-the-art performance, outperforming mainstream FR-IQA metrics.
  • Specifically, the proposed method achieved the highest SRCC (0.9286), PLCC (0.9320), and KRCC (0.7585) scores on their DHHQA database, with a significantly lower RMSE (7.2910) compared to other methods. For instance, it was approximately 0.03 higher in SRCC than the next best competitor, LPIPS.

Limitations and Future Work

  • Limitations: The proposed method is a full-reference method, meaning it requires the original, undistorted digital human head model for comparison. This can be a limitation in scenarios where only the distorted version is available.
  • Future Work: The authors suggest that their work lays the foundation for developing no-reference (NR) quality assessment methods for digital human heads, which would be more practical in many real-world applications. Further exploration into different distortion types and more complex digital human models could also be beneficial.

Summary updated

Abstract

Digital humans are attracting more and more research interest during the last decade, the generation, representation, rendering, and animation of which have been put into large amounts of effort. However, the quality assessment of digital humans has fallen behind. Therefore, to tackle the challenge of digital human quality assessment issues, we propose the first large-scale quality assessment database for three-dimensional (3D) scanned digital human heads (DHHs). The constructed database consists of 55 reference DHHs and 1,540 distorted DHHs along with the subjective perceptual ratings. Then, a simple yet effective full-reference (FR) projection-based method is proposed to evaluate the visual quality of DHHs. The pretrained Swin Transformer tiny is employed for hierarchical feature extraction and the multi-head attention module is utilized for feature fusion. The experimental results reveal that the proposed method exhibits state-of-the-art performance among the mainstream FR metrics, which can provide an effective FR-IQA index for DHHs.
BibTeX
@inproceedings{zhang2023perceptual,
  author = {Zicheng Zhang and Yingjie Zhou and Wei Sun and Xiongkuo Min and Guangtao Zhai},
  title = {{Perceptual Quality Assessment for Digital Human Heads}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095347},
}
Figure