Screen-based 3D Subjective Experiment Software

Songlin Fan, Wei Gao

Abstract

As 3D graphics such as point clouds and meshes become increasingly prevalent in both industry and academia, the need for accurate perceptual quality assessment has grown. However, the lack of specialized software for conducting 3D subjective experiments has historically complicated the construction of high-quality datasets. This paper introduces a powerful, Unity-based platform designed to streamline 3D subjective methodologies. By enabling synchronous rendering and interaction between source and impaired stimuli, the software minimizes cognitive overload for participants and provides a more ecologically valid assessment environment. Experimental validation involving 40 participants demonstrates that the platform produces highly reliable subjective quality scores, with a Spearman Rank Order Correlation Coefficient (SROCC) of 0.9632. All resources are available at https://openi.pcl.ac.cn/OpenDatasets/3DQA.

Introduction

From digital twins to the burgeoning metaverse, 3D content is transforming how we interact with digital information. To ensure a high-quality user experience, researchers must develop sophisticated compression and transmission algorithms. However, the development of these algorithms relies heavily on subjective quality assessment—the process of having humans rate the visual quality of 3D models.

Currently, researchers face a significant methodological gap. They often resort to projecting 3D models into 2D videos, which removes the viewer's ability to interact with the object, or they use general-purpose visualization tools like CloudCompare, which are not designed for formal experimentation. These "clunky" methods often lead to inconsistent results and high participant fatigue. There is a clear need for a dedicated, interactive, and repeatable 3D subjective experiment platform.

The Proposed Platform

To address these challenges, we developed a dedicated software platform using the Unity engine. The tool is designed to provide a seamless experience for both researchers and participants through several key technical features:

Direct 3D Rendering

Unlike methods that convert 3D assets into 2D proxies, our software renders PLY and OBJ files directly. This approach avoids "secondary distortions" that can occur during conversion, ensuring that participants are evaluating the actual 3D data.

Synchronous Interaction

The platform's standout feature is its ability to synchronize the movement of two models. When a participant rotates, pans, or zooms into the "source" model, the "impaired" model responds identically and simultaneously. This allows for a direct, "apples-to-apples" comparison of specific details without requiring the participant to manually align two different views.

Experiment Management Tools

The software includes built-in features to ensure experimental rigor:

  • Customizable Timers: Researchers can set specific viewing time limits.
  • Breakpoint Startup: This allows participants to pause and resume long sessions, preventing fatigue-induced errors.
  • Flexible Display Modes: Supports both synchronous and asynchronous viewing, as well as customizable background colors to test the impact of different lighting environments.

Experimental Validation

To verify the validity of the software, we organized a subjective experiment involving 40 participants.

Dataset Construction

We created a benchmark using five high-quality source point cloud models: Rose, Statue, Girl, Sneaker, and Man. Each model contained over 600,000 points with color and normal attributes. From these, we generated 40 impaired stimuli using the AVS Point Cloud Reference Model (PCRM), applying eight different combinations of geometry and attribute compression to ensure a wide range of visual quality.

Methodology

Participants were tasked with rating the models based on their perceived quality. This resulted in a dataset of 2,000 individual quality judgments. To test for reliability, we split the participants into two groups and compared their Mean Opinion Scores (MOS).

Key Results and Insights

High Reliability and Consistency

The experimental analyses demonstrated that the software produces remarkably consistent results. The correlation between the two participant groups was extremely high:

  • SROCC: 0.9632
  • PLCC: 0.9962

Furthermore, the MOS correlated perfectly with the levels of compression applied; as geometry or attribute data was removed, the scores dropped predictably, confirming that the software accurately captures human perception of 3D distortions.

Reducing Cognitive Overload

By synchronizing the movement of the models, the software significantly reduces the mental effort required to compare 3D objects. Participants can focus entirely on spotting visual artifacts rather than struggling with navigation controls. This makes the resulting scores more accurate and reduces the likelihood of participant fatigue during long sessions.

Ecological Validity

Because the software allows for free rotation and interaction, it mimics how users actually engage with 3D applications in real-world scenarios. This makes the data more relevant for developers building real-time 3D applications compared to data derived from static 2D snapshots.

Conclusion and Future Work

The proposed software offers a robust solution for the 3D graphics quality assessment community and has already been accepted as a reference tool by the Audio Video coding Standard (AVS). While the current version is screen-based, future work will explore adapting these synchronous interaction principles for Virtual Reality (VR) and Augmented Reality (AR) environments. Additionally, we plan to expand the software's capabilities to evaluate a broader range of distortions beyond compression, such as sensor noise and simplification artifacts, and explore the potential for web-based crowdsourcing.

Summary updated

Abstract

Recently, widespread 3D graphics (e.g., point clouds and meshes) have drawn considerable efforts from academia and industry to assess their perceptual quality by conducting subjective experiments. However, lacking a handy software for 3D subjective experiments complicates the construction of 3D graphics quality assessment datasets, thus hindering the prosperity of relevant fields. In this paper, we develop a powerful platform with which users can flexibly design their 3D subjective methodologies and build high-quality datasets, easing a broad spectrum of 3D graphics subjective quality study. To accurately illustrate the perceptual quality differences of 3D stimuli, our software can simultaneously render the source stimulus and impaired stimulus and allows both stimuli to respond synchronously to viewer interactions. Compared with amateur 3D visualization tool-based or image/video rendering-based schemes, our approach embodies typical 3D applications while minimizing cognitive overload during subjective experiments. We organized a subjective experiment involving 40 participants to verify the validity of the proposed software. Experimental analyses demonstrate that subjective tests on our software can produce reasonable subjective quality scores of 3D models. All resources in this paper can be found at https://openi.pcl.ac.cn/OpenDatasets/3DQA.
BibTeX
@inproceedings{fan2023screen,
  author = {Songlin Fan and Wei Gao},
  title = {{Screen-based 3D Subjective Experiment Software}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3613457},
}
Figure