Abstract
Although 3D generated content (3DGC) offers advantages in reducing production costs and accelerating design timelines, its quality often falls short when compared to 3D professionally generated content (PGC). Common quality issues frequently affect 3DGC, highlighting the importance of timely and effective quality assessment. To address existing gaps, this paper introduces a novel 3DGC quality assessment dataset, 3DGCQA, built using 7 representative Text-to-3D generation methods. Using 50 fixed prompts, the researchers created 313 textured meshes. Subjective quality assessment revealed significant variation across methods, while objective tests exposed limitations in existing algorithms, underscoring the need for specialized 3D quality assessment tools. The dataset is open-sourced at https://github.com/zyj-2000/3DGCQA.
Introduction
While AI can now generate 3D models from simple text prompts, the results are often inconsistent. Unlike professionally generated content (PGC), which maintains high standards of structural integrity and texture alignment, AI-generated content (AIGC) frequently suffers from bizarre geometric distortions and blurry textures.
To advance generative models, precise quality measurement is essential. However, most existing 3D quality assessment (3DQA) tools were designed for scanned real-world objects or compressed point clouds, making them ill-suited for the unique artifacts produced by AI. The 3DGCQA database addresses this by providing a standardized set of AI-generated 3D models paired with human quality scores.
Methodology and Dataset Construction
The researchers utilized 7 representative Text-to-3D methods—including DreamFusion, Magic3D, and HiFA—to generate models based on 50 diverse prompts across 10 categories (e.g., animals, tools, humans). This resulted in 313 textured meshes featuring various artistic styles like "Japanese ink style" or "cyberpunk."
During construction, the team identified 6 common distortion categories unique to 3DGC:
- Multifaceted Repetition: The "Janus problem," where an object has multiple faces or heads.
- Depth Error: Incorrect spatial relationships between parts.
- Roughness: Unintended surface noise on the mesh.
- Misalignment: Texture and geometry failing to line up.
- Geometry Loss: Missing components of the intended object.
- Geometry Redundancy: Extra, floating bits of mesh.
Subjective and Objective Quality Analysis
Subjective scoring was conducted by 40 human evaluators who rated the models on alignment (prompt matching) and overall quality. While most models aligned well with prompts, visual quality scores averaged a mediocre 2.5 out of 5. Notably, human models received the lowest scores due to the high complexity of human textures and the strict geometric symmetry expected by the human eye.
Magic3D emerged as the top-performing generator in terms of visual quality. Furthermore, the choice of the underlying Text-to-Image (T2I) model proved critical; methods utilizing DeepFloyd IF generally yielded higher-quality 3D outputs than those using Stable Diffusion.
In benchmarking objective metrics, the Deep Bilinear Convolutional Neural Network (DBCNN) was the top performer with a Spearman Rank Correlation (SRCC) of 0.5381. Traditional "naturalness" metrics like BRISQUE or NIQE performed poorly (SRCC ~0.20), as they are designed for photos and fail to capture the specific artifacts of AI-generated 3D meshes. This highlights a significant "2D-3D gap," where metrics based on 2D snapshots lose vital geometric information perceived by humans.
Conclusion and Future Work
The 3DGCQA dataset provides a vital foundation for the development of 3D-aware assessment tools. While 313 models is a significant start, larger datasets are needed to train robust deep-learning models. Future research should expand beyond Text-to-3D to include Image-to-3D and Video-to-3D content. Ultimately, the field requires new frameworks that directly analyze 3D data structures, such as meshes or Gaussian Splats, to better capture the nuances of 3D quality.
Summary updated