LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM

Zicheng Zhang, Haoning Wu, Yingjie Zhou, Chunyi Li, Wei Sun, Chaofeng Chen, Xiongkuo Min, Xiaohong Liu, Weisi Lin, Guangtao Zhai

Abstract

Point Cloud Quality Assessment (PCQA) is critical for 3D applications such as autonomous driving, VR/AR, and digital heritage preservation. However, accurately predicting human perception of 3D data remains a significant challenge due to the complexity of geometric distortions. This paper introduces LMM-PCQA, the first framework to successfully leverage Large Multi-modality Models (LMMs) for no-reference 3D point cloud quality assessment. By integrating the high-level visual "common sense" of LMMs with low-level geometric structural features, LMM-PCQA achieves state-of-the-art performance across multiple benchmarks, effectively bridging the gap between 2D visual perception and 3D geometric reality.

Introduction

Point clouds, which represent 3D objects as collections of millions of points, are the backbone of modern 3D technologies. During acquisition, compression, and transmission, these data are often subject to noise and degradation. Assessing the visual quality of these objects is notoriously difficult; traditional methods often require a perfect reference model or rely on handcrafted mathematical descriptors that fail to capture human visual nuances. While LMMs have revolutionized quality assessment for 2D media, their potential for 3D point clouds remained untapped. LMM-PCQA addresses this by projecting 3D data into 2D space, allowing the model to utilize pre-trained visual knowledge for 3D assessment.

Methodology

The LMM-PCQA framework employs a dual-stream architecture to capture both visual and structural quality indicators.

Visual Stream: LMM Evaluation

To leverage the 2D visual knowledge of LMMs, 3D point clouds are rendered into six orthogonal 2D projections. These images are processed by a fine-tuned LMM, mPLUG-Owl2.

  • Instruction Tuning: Instead of direct numerical regression, the model is trained to classify quality using five qualitative adjectives: Excellent, Good, Fair, Poor, and Bad. This approach aligns the model's output with human linguistic evaluation.
  • Probabilistic Scoring: The model extracts logits (probabilities) for these quality terms, providing a nuanced quality representation that is more effective than a single label.

Structural Stream: Geometric Features

Recognizing that 2D projections may lose fine-grained 3D details, a parallel stream extracts multi-scale structural features directly from the 3D coordinates. Using k-nearest neighbors (k-NN), the model calculates geometric properties such as linearity and planarity to capture local structural integrity.

Quality Regression

The visual probabilities from the LMM and the geometric features from the structural stream are fused and passed through a Support Vector Regressor (SVR) to output a final, continuous quality score.

Datasets

The model was validated on three major benchmarks:

  • SJTU-PCQA: 378 distorted point clouds with 7 distortion types.
  • Waterloo Point Cloud (WPC): 740 distorted point clouds, including G-PCC and V-PCC compression.
  • WPC2.0: 400 point clouds focused on video-based compression (V-PCC) degradations.

Key Results and Insights

LMM-PCQA achieved top-tier performance across all tested databases, often surpassing even Full-Reference methods:

  • State-of-the-Art Performance: On the SJTU-PCQA dataset, the model achieved a Spearman Rank Correlation Coefficient (SRCC) of 0.9376, significantly outperforming previous no-reference methods.
  • LMM "Common Sense": A core finding is that LMMs possess robust knowledge of visual quality from their 2D training. Projecting 3D clouds into 2D allows this knowledge to be transferred to the 3D domain.
  • Viewpoint Optimization: Performance analysis indicated that quality assessment improves as the number of projections increases, reaching an optimal balance at six orthogonal views (a "cube-map" representation).
  • Generalization: Cross-database testing demonstrated the model's robustness, proving it learns general quality concepts rather than dataset-specific artifacts.

Conclusion and Future Work

LMM-PCQA represents a significant step forward in no-reference point cloud quality assessment. By combining qualitative LMM evaluation with direct geometric analysis, the framework effectively models human perception of 3D data. While the current reliance on 2D projections is a limitation, future work will explore LMMs that can process 3D coordinates directly and investigate deeper fusion techniques for visual and structural features.

Summary updated

Abstract

Although large multi-modality models (LMMs) have seen extensive exploration and application in various quality assessment studies, their integration into Point Cloud Quality Assessment (PCQA) remains unexplored. Given LMMs' exceptional performance and robustness in low-level vision and quality assessment tasks, this study aims to investigate the feasibility of imparting PCQA knowledge to LMMs through text supervision. To achieve this, we transform quality labels into textual descriptions during the fine-tuning phase, enabling LMMs to derive quality rating logits from 2D projections of point clouds. To compensate for the loss of perception in the 3D domain, structural features are extracted as well. These quality logits and structural features are then combined and regressed into quality scores. Our experimental results affirm the effectiveness of our approach, showcasing a novel integration of LMMs into PCQA that enhances model understanding and assessment accuracy. We hope our contributions can inspire subsequent investigations into the fusion of LMMs with PCQA, fostering advancements in 3D visual quality analysis and beyond.
BibTeX
@inproceedings{zhang2024lmm,
  author = {Zicheng Zhang and Haoning Wu and Yingjie Zhou and Chunyi Li and Wei Sun and Chaofeng Chen and Xiongkuo Min and Xiaohong Liu and Weisi Lin and Guangtao Zhai},
  title = {{LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3680946},
}
Figure