3DGCQA: A Quality Assessment Database for 3D AI-Generated Contents

Yingjie Zhou, Zicheng Zhang, Farong Wen, Jun Jia, Yanwei Jiang, Xiaohong Liu, Xiongkuo Min, Guangtao Zhai

TL;DR

Researchers have introduced 3DGCQA, the first specialized database designed to evaluate the quality of 3D AI-generated content (3DGC). By benchmarking seven state-of-the-art Text-to-3D generation methods, the study reveals that current objective quality metrics struggle to keep up with human perception, highlighting a critical need for more sophisticated, 3D-aware assessment tools.

What Problem Does This Paper Address?

While AI can now generate 3D models from simple text prompts, the results are often "hit or miss." Unlike professionally generated content (PGC), AI-generated meshes frequently suffer from bizarre geometric distortions, blurry textures, or a total lack of structural integrity.

To improve these generative models, we need a way to measure "quality." However, most existing 3D quality assessment (3DQA) tools were designed for scanned real-world objects or compressed point clouds, not the unique, often surreal artifacts produced by AI. There has been a lack of standardized datasets that pair AI-generated 3D models with human quality scores—until now.

Methods

The authors took a systematic approach to building and testing their database:

  1. Generation: They used 7 representative Text-to-3D methods (including DreamFusion, Magic3D, and HiFA) to generate models based on 50 diverse prompts.
  2. Taxonomy of Failure: They identified 6 common distortion categories unique to 3DGC:
  3. Multifaceted Repetition: The "Janus problem" where an object has multiple faces or heads.
  4. Depth Error: Incorrect spatial relationships.
  5. Roughness: Unintended "noise" on the mesh surface.
  6. Misalign: Texture and geometry not lining up.
  7. Geometry Loss: Missing parts of the object.
  8. Geometry Redundancy: Extra, floating bits of mesh.
  9. Subjective Scoring: 40 human evaluators rated the 313 resulting models on two scales: alignment (how well it matches the prompt) and overall quality.
  10. Objective Benchmarking: They tested 8 existing image-based quality assessment algorithms to see if they could predict the human scores.

Datasets

The paper introduces the 3DGCQA dataset:

  • Content: 313 textured meshes.
  • Source: Generated from 50 prompts across 10 categories (tools, animals, humans, etc.).
  • Diversity: Includes various artistic styles (e.g., "Japanese ink style," "cyberpunk") and "best quality" constraints.
  • Availability: The dataset is open-sourced on GitHub.

Key Insights and Ideas

  • The "Human" Hurdle: Models of humans consistently received the lowest quality scores. The authors attribute this to the high complexity of human textures and the strict geometric symmetry our eyes expect, making AI failures in this category particularly jarring.
  • Magic3D Leads the Pack: Among the tested generators, Magic3D achieved the best overall performance in terms of visual quality.
  • T2I Models Matter: The quality of the underlying Text-to-Image (T2I) model (like Stable Diffusion vs. DeepFloyd IF) significantly impacts the final 3D result. DeepFloyd IF generally led to higher-quality 3D outputs.
  • The 2D-3D Gap: Most current "objective" metrics work by looking at 2D projections (snapshots) of the 3D model. The authors found these are insufficient because they lose vital geometric and depth information that humans perceive when viewing a 3D object.

Key Results

  • Human Agreement: The subjective scores (Mean Opinion Scores or MOS) showed that while most models aligned well with prompts, the actual visual quality hovered around a mediocre 2.5 out of 5.
  • Metric Performance: The DBCNN (Deep Bilinear Convolutional Neural Network) was the top-performing objective metric, achieving a Spearman Rank Correlation (SRCC) of 0.5381.
  • The Failure of Traditional Metrics: Hand-crafted features (like BRISQUE or NIQE) performed poorly (SRCCs around 0.20), proving that "naturalness" metrics for photos don't translate well to the artifacts found in AI-generated 3D meshes.

Limitations and Future Work

  • Dataset Scale: While 313 models is a great start, it is relatively small for training robust deep-learning models.
  • Scope: The study focused exclusively on Text-to-3D generation. Future work could expand to Image-to-3D or Video-to-3D content.
  • Need for 3D-Native Metrics: The results underscore that we cannot rely on 2D snapshots forever. The authors call for new frameworks that directly analyze 3D data structures (like meshes or Gaussian Splats) to better capture the nuances of 3D quality.

Summary updated

Abstract

Although 3D generated content (3DGC) offers advantages in reducing production costs and accelerating design timelines, its quality often falls short when compared to 3D professionally generated content. Common quality issues frequently affect 3DGC, highlighting the importance of timely and effective quality assessment. Such evaluations not only ensure a higher standard of 3DGCs for end-users but also provide critical insights for advancing generative technologies. To address existing gaps in this domain, this paper introduces a novel 3DGC quality assessment dataset, 3DGCQA, built using 7 representative Text-to-3D generation methods. During the dataset’s construction, 50 fixed prompts are utilized to generate contents across all methods, resulting in the creation of 313 textured meshes that constitute the 3DGCQA dataset. The visualization intuitively reveals the presence of 6 common distortion categories in the generated 3DGCs. To further explore the quality of the 3DGCs, subjective quality assessment is conducted by evaluators, whose ratings reveal significant variation in quality across different generation methods. Additionally, several objective quality assessment algorithms are tested on the 3DGCQA dataset. The results expose limitations in the performance of existing algorithms and underscore the need for developing more specialized quality assessment methods. To provide a valuable resource for future research and development in 3D content generation and quality assessment, the dataset has been open-sourced in https://github.com/zyj-2000/3DGCQA.
BibTeX
@inproceedings{zhou2025dgcqa,
  author = {Yingjie Zhou and Zicheng Zhang and Farong Wen and Jun Jia and Yanwei Jiang and Xiaohong Liu and Xiongkuo Min and Guangtao Zhai},
  title = {{3DGCQA: A Quality Assessment Database for 3D AI-Generated Contents}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889921},
}

Loading related content...

Figure