Abstract
As 3D graphics such as point clouds and meshes become increasingly prevalent in both industry and academia, the need for accurate perceptual quality assessment has grown. However, the lack of specialized software for conducting 3D subjective experiments has historically complicated the construction of high-quality datasets. This paper introduces a powerful, Unity-based platform designed to streamline 3D subjective methodologies. By enabling synchronous rendering and interaction between source and impaired stimuli, the software minimizes cognitive overload for participants and provides a more ecologically valid assessment environment. Experimental validation involving 40 participants demonstrates that the platform produces highly reliable subjective quality scores, with a Spearman Rank Order Correlation Coefficient (SROCC) of 0.9632. All resources are available at https://openi.pcl.ac.cn/OpenDatasets/3DQA.
Introduction
From digital twins to the burgeoning metaverse, 3D content is transforming how we interact with digital information. To ensure a high-quality user experience, researchers must develop sophisticated compression and transmission algorithms. However, the development of these algorithms relies heavily on subjective quality assessment—the process of having humans rate the visual quality of 3D models.
Currently, researchers face a significant methodological gap. They often resort to projecting 3D models into 2D videos, which removes the viewer's ability to interact with the object, or they use general-purpose visualization tools like CloudCompare, which are not designed for formal experimentation. These "clunky" methods often lead to inconsistent results and high participant fatigue. There is a clear need for a dedicated, interactive, and repeatable 3D subjective experiment platform.
The Proposed Platform
To address these challenges, we developed a dedicated software platform using the Unity engine. The tool is designed to provide a seamless experience for both researchers and participants through several key technical features:
Direct 3D Rendering
Unlike methods that convert 3D assets into 2D proxies, our software renders PLY and OBJ files directly. This approach avoids "secondary distortions" that can occur during conversion, ensuring that participants are evaluating the actual 3D data.
Synchronous Interaction
The platform's standout feature is its ability to synchronize the movement of two models. When a participant rotates, pans, or zooms into the "source" model, the "impaired" model responds identically and simultaneously. This allows for a direct, "apples-to-apples" comparison of specific details without requiring the participant to manually align two different views.
Experiment Management Tools
The software includes built-in features to ensure experimental rigor:
- Customizable Timers: Researchers can set specific viewing time limits.
- Breakpoint Startup: This allows participants to pause and resume long sessions, preventing fatigue-induced errors.
- Flexible Display Modes: Supports both synchronous and asynchronous viewing, as well as customizable background colors to test the impact of different lighting environments.
Experimental Validation
To verify the validity of the software, we organized a subjective experiment involving 40 participants.
Dataset Construction
We created a benchmark using five high-quality source point cloud models: Rose, Statue, Girl, Sneaker, and Man. Each model contained over 600,000 points with color and normal attributes. From these, we generated 40 impaired stimuli using the AVS Point Cloud Reference Model (PCRM), applying eight different combinations of geometry and attribute compression to ensure a wide range of visual quality.
Methodology
Participants were tasked with rating the models based on their perceived quality. This resulted in a dataset of 2,000 individual quality judgments. To test for reliability, we split the participants into two groups and compared their Mean Opinion Scores (MOS).
Key Results and Insights
High Reliability and Consistency
The experimental analyses demonstrated that the software produces remarkably consistent results. The correlation between the two participant groups was extremely high:
- SROCC: 0.9632
- PLCC: 0.9962
Furthermore, the MOS correlated perfectly with the levels of compression applied; as geometry or attribute data was removed, the scores dropped predictably, confirming that the software accurately captures human perception of 3D distortions.
Reducing Cognitive Overload
By synchronizing the movement of the models, the software significantly reduces the mental effort required to compare 3D objects. Participants can focus entirely on spotting visual artifacts rather than struggling with navigation controls. This makes the resulting scores more accurate and reduces the likelihood of participant fatigue during long sessions.
Ecological Validity
Because the software allows for free rotation and interaction, it mimics how users actually engage with 3D applications in real-world scenarios. This makes the data more relevant for developers building real-time 3D applications compared to data derived from static 2D snapshots.
Conclusion and Future Work
The proposed software offers a robust solution for the 3D graphics quality assessment community and has already been accepted as a reference tool by the Audio Video coding Standard (AVS). While the current version is screen-based, future work will explore adapting these synchronous interaction principles for Virtual Reality (VR) and Augmented Reality (AR) environments. Additionally, we plan to expand the software's capabilities to evaluate a broader range of distortions beyond compression, such as sensor noise and simplification artifacts, and explore the potential for web-based crowdsourcing.
Summary updated