TL;DR
Researchers have developed a new Unity-based software platform specifically designed for conducting subjective quality assessments of 3D graphics like point clouds and meshes. By allowing users to interact with source and distorted models side-by-side in real-time with synchronized movements, the tool significantly reduces the "cognitive overload" that typically plagues 3D quality studies.
What Problem Does This Paper Address?
As 3D content—from digital twins to metaverse assets—becomes more prevalent, we need better ways to compress and transmit this data without ruining the visual experience. To build better compression algorithms, we first need to know how humans perceive 3D quality.
Currently, researchers face a dilemma: they either project 3D models into 2D videos (which sacrifices the viewer's freedom to look around) or they use manual visualization tools like CloudCompare (which are clunky for formal experiments and lead to inconsistent results). There is a distinct lack of "handy" software that allows for interactive, controlled, and repeatable 3D subjective experiments.
Methods
The authors built a dedicated platform using the Unity engine. Key technical features include:
- Direct 3D Rendering: Unlike methods that convert 3D assets into 2D images or videos, this software renders PLY and OBJ files directly. This avoids "secondary distortions" that might occur during a conversion process.
- Synchronous Interaction: This is the platform's standout feature. When a participant rotates or zooms into the "source" model, the "impaired" model responds identically and simultaneously. This allows for a direct, "apples-to-apples" comparison of specific details.
- Experiment Management: The tool includes built-in features for researchers, such as a customizable timer, "breakpoint startup" (allowing participants to pause and resume long sessions), and the ability to switch background colors to see how different lighting/backdrops affect perceived quality.
- Flexible Methodology: It supports various display modes (synchronous vs. asynchronous) and allows researchers to set specific viewing time limits and scoring scales.
Datasets
To validate the software, the authors created a small but diverse benchmark:
- Source Stimuli: Five high-quality point cloud models (Rose, Statue, Girl, Sneaker, and Man), each containing over 600,000 points with color and normal attributes.
- Impaired Stimuli: They generated 40 distorted versions of these models using the AVS Point Cloud Reference Model (PCRM). They applied eight different combinations of geometry and attribute compression to ensure a wide range of visual quality.
- Evaluation Data: They recruited 40 participants to rate these models, resulting in a dataset of 2,000 individual quality judgments.
Key Insights and Ideas
- Reducing Cognitive Overload: The authors argue that the biggest hurdle in 3D quality assessment isn't the rendering—it's the human brain. By synchronizing the movement of two models, the software removes the mental effort required to manually align two different views, allowing the participant to focus entirely on spotting visual artifacts.
- The "Interactive" Advantage: By allowing participants to rotate models freely, the experiment mimics how users actually interact with 3D applications in the real world, making the resulting Mean Opinion Scores (MOS) more ecologically valid than static 2D snapshots.
- Standardization: The software has already been accepted as a reference tool by the Audio Video coding Standard (AVS), suggesting it may become a cornerstone for future 3D codec development.
Key Results
The authors performed a cross-validation by splitting their 40 participants into two groups. The results showed remarkable consistency:
- Correlation: The correlation between the two groups' scores was extremely high, with a Spearman Rank Order Correlation Coefficient (SROCC) of 0.9632 and a Pearson Linear Correlation Coefficient (PLCC) of 0.9962.
- Reliability: The Mean Opinion Scores (MOS) correlated perfectly with the levels of compression applied; as geometry or attribute data was removed, the scores dropped predictably.
- Efficiency: The "breakpoint" and "timer" features allowed for the successful management of a large-scale subjective test without significant participant fatigue or data loss.
Limitations and Future Work
While the paper presents a robust tool, a few areas remain open for exploration:
- Platform Scope: The current version is "screen-based." Future iterations might need to adapt these synchronous interaction principles for Virtual Reality (VR) or Augmented Reality (AR) headsets, where 3D perception is even more complex.
- Distortion Types: The validation focused primarily on compression artifacts. Future studies could use the software to evaluate other types of 3D noise, such as sensor errors or simplification artifacts.
- Scalability: While the software is flexible, the paper focuses on lab-based studies. Adapting this for web-based crowdsourcing (where hardware capabilities vary wildly) would be a significant next step.
Summary updated