Cross-Modality Feature Fusion Network for Few-Shot 3D Point Cloud Classification

Minmin Yang, Jiajing Chen, Senem VelipasalarWACV 2023

TL;DR

This paper introduces a novel network that fuses depth images and 3D point cloud data to improve few-shot 3D object classification, especially for objects with missing parts. Their approach uses a cross-modality feature fusion network with a unique Support-Query Mutual Attention (sqMA) module and a pyramid pooling operation to enhance feature representation and classification accuracy.

What Problem Does This Paper Address?

Few-shot 3D point cloud classification is a challenging task because real-world 3D data often suffers from occlusions, noise, and missing points, making it difficult to recognize objects with only a few examples. Traditional deep learning methods for 3D data struggle with these imperfections. While depth images can sometimes compensate for missing point cloud data, they may lack the detailed shape information present in point clouds. This paper aims to bridge this gap by effectively combining the strengths of both modalities to achieve more robust and accurate few-shot classification.

Methods

The core of the proposed method is a Cross-Modality Feature Fusion Network. This network processes two types of input:

  • Depth Images: Generated by projecting the 3D point cloud onto six orthogonal planes. These are processed by a standard ResNet-18 backbone.
  • Raw Point Clouds: Processed by a DGCNN backbone.

Key novel components include:

  • Pyramid Pooling Operation: Inspired by Horizontal Pyramid Mapping, this operation replaces traditional max/average pooling. It divides feature maps into strips of varying scales and applies global pooling to each strip, aiming to retain more global and local features and reduce information loss.
  • Support-Query Mutual Attention (sqMA) Module: This module is designed to adaptively update support and query features by considering their mutual correlation. Unlike previous methods that might use prototype features, sqMA directly uses support features to update query features and vice-versa, allowing for a more nuanced learning of inter-example relationships.

Datasets

The paper evaluates its method on three standard 3D point cloud datasets:

  • ModelNet40: A widely used dataset of 40 object categories, consisting of clean CAD models.
  • ModelNet40-C: A corrupted version of ModelNet40, designed to simulate real-world scenarios with 15 types of realistic corruptions.
  • ScanObjectNN: A dataset collected from real-world indoor scenes, which is more challenging due to natural occlusions and missing points.

Key Insights and Ideas

  • Complementary Modalities: Depth images and point clouds offer complementary information. Depth images are more robust to missing points, while point clouds provide finer shape details. Combining them leads to significantly better performance.
  • Attention for Correlation: The sqMA module is crucial for handling intra-class variations. By explicitly modeling the correlation between support and query features, it allows the network to adaptively learn which support examples are most relevant for classifying a given query.
  • Preserving Features: Standard pooling operations can discard valuable information. The proposed pyramid pooling method effectively retains more detailed features, leading to improved classification.
  • Robustness to Missing Data: The method shows strong performance on datasets with corrupted or missing data (ModelNet40-C and ScanObjectNN), highlighting its practical applicability.

Key Results

The proposed method consistently outperforms several state-of-the-art baselines across all three datasets.

  • On the challenging ScanObjectNN dataset, the proposed method achieved a significant improvement of up to 9.99% in 5-way 5-shot classification compared to the second-best performer.
  • Ablation studies demonstrated that:
  • Combining depth images and point clouds yields better results than using either modality alone.
  • The sqMA module provides a notable accuracy increase of 2.07% in 1-shot and 1.45% in 5-shot settings.
  • The pyramid pooling operation improves accuracy by 2.65% in 1-shot and 3.3% in 5-shot settings compared to standard max-pooling.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of two backbones (ResNet-18 and DGCNN) and the attention mechanism likely increase computational complexity compared to single-modality methods.
  • Future Work: The paper suggests that their method's effectiveness on real-world datasets like ScanObjectNN, which inherently have missing data, opens avenues for further exploration in more challenging, unconstrained environments. The authors also mention that the sqMA module could potentially be integrated into other baseline models to improve their performance.

Summary updated

BibTeX
@inproceedings{yang2023cross,
  author = {Minmin Yang and Jiajing Chen and Senem Velipasalar},
  title = {{Cross-Modality Feature Fusion Network for Few-Shot 3D Point Cloud Classification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure