MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding

Hai-Tao Yu, Mofei Song

TL;DR

This paper introduces MM-Point, a novel self-supervised learning framework that leverages multi-view 2D images to significantly improve 3D point cloud understanding. By introducing multi-modal contrastive learning strategies and innovative augmentation techniques, MM-Point achieves state-of-the-art performance on various downstream tasks, demonstrating its effectiveness in learning robust 3D representations.

What Problem Does This Paper Address?

The field of 3D perception is rapidly growing, with point clouds being a fundamental representation for tasks like object classification, detection, and segmentation. However, obtaining labeled 3D data is expensive and time-consuming, making self-supervised learning a crucial area of research. While existing methods often rely on single-view 2D information or limited multi-view approaches, this paper addresses the challenge of effectively utilizing the rich, complementary information present in multiple 2D views of a 3D object to enhance self-supervised 3D point cloud representation learning. The core problem is how to bridge the gap between 2D and 3D representations in a way that maximizes mutual information and avoids negative transfer, leading to more robust and generalizable 3D features.

Methods

MM-Point proposes a multi-modal self-supervised learning framework that integrates 3D point cloud data with rendered 2D multi-view images. The core of the method lies in two main contrastive learning strategies:

  • Intra-modal Contrastive Learning: This focuses on learning representations within the 3D point cloud modality itself. Augmented versions of the same point cloud are used as positive pairs, while different point clouds serve as negative pairs.
  • Inter-modal Contrastive Learning: This is the key innovation, where 3D point clouds are aligned with their corresponding rendered 2D multi-view images. The goal is to learn a shared embedding space where similar 3D objects and their views are close, and dissimilar ones are far apart.

To enhance this inter-modal learning, MM-Point introduces:

  • Multi-MLP Strategy: This involves constructing multi-level feature spaces using multiple MLPs. This allows for contrasting 2D and 3D features in different dimensional spaces, helping to alleviate consistency conflicts between various views and the 3D object.
  • Multi-level Augmentation Strategy: This strategy incrementally applies augmentations to the 2D multi-view images. By varying the intensity and type of augmentation, the model is encouraged to learn invariance across different levels of transformation, leading to more robust representations. This is achieved by defining a pipeline of transformations $T_1, T_2, \dots, T_m$, where each $T_i$ applies a specific set of augmentations.

The overall objective is to maximize the mutual information between the 3D point cloud and its multiple 2D views.

Datasets

The paper utilizes the following datasets:

  • ShapeNet: Used for pre-training the MM-Point framework. It's a large-scale 3D shape dataset from which 2D multi-views are rendered.
  • ModelNet40 and ModelNet10: Standard synthetic 3D object datasets used for evaluating the learned representations in classification tasks.
  • ScanObjectNN: A real-world 3D object dataset used to assess the model's performance on more challenging, realistic data.
  • S3DIS: A dataset for 3D semantic segmentation, used to evaluate the fine-grained feature learning capabilities.
  • ShapeNetPart: Used for 3D part segmentation, another task that requires understanding fine-grained geometric details.

Key Insights and Ideas

  • Leveraging Multi-View Richness: The paper strongly advocates that multiple 2D views of an object offer significantly richer self-supervised signals for 3D understanding than single views. The complementary information across different angles is crucial.
  • Bridging 2D-3D Gap with Contrast: The core idea of using contrastive learning to align 3D point clouds with their rendered 2D views is a powerful way to transfer visual knowledge. The paper emphasizes that this alignment needs to be carefully designed to avoid negative transfer.
  • Multi-Level Feature Learning is Key: The Multi-MLP strategy highlights that simply projecting features into a single space might not be optimal. Creating multi-level feature spaces allows for a more nuanced interaction between modalities.
  • Augmentation as a Learning Signal: The multi-level augmentation strategy demonstrates that controlled, hierarchical augmentation can act as a powerful tool for learning invariance and improving robustness, rather than just being a data augmentation technique.
  • Transferability is Paramount: The success of MM-Point across various downstream tasks (classification, segmentation, few-shot learning) underscores the quality and generalizability of the learned 3D representations.

Key Results

MM-Point achieves state-of-the-art (SOTA) performance on several benchmarks:

  • 3D Object Classification:
    • ModelNet40: Achieves a peak accuracy of 92.4%.
  • ModelNet10: Achieves a peak accuracy of 95.4%.
  • ScanObjectNN: Achieves a top accuracy of 87.8%, outperforming existing self-supervised methods by a significant margin (e.g., +6.1% over CrossPoint).
  • 3D Part Segmentation: Achieves an OA of 88.7% and mIoU of 59.1% on S3DIS, showing a +4.2% improvement in mIoU over the baseline.
  • 3D Semantic Segmentation: Achieves an OA of 94.5% and mIoU of 85.7% on S3DIS.
  • Few-Shot Classification: Demonstrates substantial improvements on both ModelNet40 and ScanObjectNN, achieving SOTA results across different N-way K-shot settings.

The paper also includes ablation studies showing that:

  • Using more than one 2D view consistently improves performance, with an optimal number of views around 4.
  • The Multi-MLP strategy and Multi-level Augmentation strategy are crucial for achieving the best results.

Limitations and Future Work

  • Number of Views: While the paper shows performance improves with more views, it also suggests an optimal number of views (around 4) exists. The exact trade-off and how to select the optimal number of views for different scenarios could be further explored.
  • Computational Cost: While self-supervised, the multi-modal nature and multiple augmentations might introduce higher computational costs compared to single-modal methods.
  • Future Work: The authors suggest that their framework can be extended to other 3D tasks and that further exploration of multi-modal fusion strategies could yield even better results. The code for MM-Point is made publicly available.

Summary updated

BibTeX
@inproceedings{yu2024mm,
  author = {Hai-Tao Yu and Mofei Song},
  title = {{MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28501},
}
Figure