Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction

Zhizhong Han, Xiyang Wang, Yu-Shen Liu, Matthias Zwicker

Abstract

Unsupervised feature learning for 3D point clouds is a critical challenge in computer vision, aiming to extract meaningful representations without extensive manual labeling. Traditional methods often rely on global self-reconstruction, which may neglect fine-grained local geometry. We propose the Multi-Angle Point Cloud Variational Auto-Encoder (MAP-VAE), which leverages a novel "half-to-half" prediction task across multiple viewpoints. By splitting a point cloud into front and back halves using geodesic distances and training the network to predict the hidden portions, MAP-VAE learns a comprehensive representation of 3D shapes. Our approach demonstrates superior performance on classification and segmentation benchmarks, rivaling supervised methods in some categories.

Introduction

Supervised learning has driven significant progress in 3D point cloud analysis for applications like autonomous driving and robotics. However, the dependency on large-scale labeled datasets remains a bottleneck. Unsupervised learning offers a path forward, but existing global reconstruction-based methods often fail to capture subtle local features. To address this, we introduce a framework that forces the model to understand the spatial relationships between different parts of a shape by predicting missing geometry from multiple angles, effectively playing a game of "geometric hide-and-seek."

Methodology

The MAP-VAE framework is built around three primary components: multi-angle splitting, feature aggregation, and a dual-task output consisting of reconstruction and prediction.

Multi-Angle Geodesic Splitting

Unlike simple Euclidean splitting, which can produce fragmented patches, we utilize geodesic distance (distance along the surface) to partition the point cloud into a "front half" (visible) and a "back half" (hidden) from various viewpoints. This ensures that the halves are semantically and geometrically continuous, facilitating more effective learning of local structures.

The multi-angle geodesic splitting process, showing how a 3D model is partitioned from multiple angles.

Model Architecture

The architecture consists of three specialized branches:

  1. Aggregation Branch (A): This branch uses a global encoder for the full shape and a local encoder for the sequence of front halves. An Aggregation RNN then processes these features to capture spatial dependencies across the different views.
  2. Prediction Branch (P): This implements the "half-to-half" task. The model takes the features of the front halves and attempts to predict the features of the corresponding hidden back halves, requiring a deep understanding of 3D geometry.
  3. Reconstruction Branch (R): A standard VAE component that ensures the model can reconstruct the entire original point cloud from its learned latent space, facilitating shape generation.
Overview of the MAP-VAE architecture, highlighting the Aggregation, Reconstruction, and Prediction branches.

Experimental Evaluation

We evaluated MAP-VAE on the gold standards of 3D deep learning: ModelNet10/40 and the ShapeNet Part dataset.

  • Classification: MAP-VAE achieved a state-of-the-art accuracy of 90.15% on ModelNet40 among unsupervised methods.
  • Segmentation: The model significantly outperformed competitors in part segmentation across almost all categories (e.g., Aero, Chair, Car), proving that its learned features capture intricate local details better than global-only methods.
  • Shape Completion: Qualitative results demonstrate the model's ability to "fill in the blanks" for partial point clouds, producing sharper edges and more accurate geometry.

Key Insights

The success of MAP-VAE stems from the realization that predicting missing parts is a more challenging and semantically rich task than simple compression. By enforcing a variational constraint, the model learns a smooth, continuous latent space suitable for both analysis and generation. Furthermore, the use of geodesic distance is crucial for maintaining the structural integrity of the input halves during training, preventing the "shattered" patches common in Euclidean-based splitting.

Conclusion

MAP-VAE provides a robust framework for unsupervised 3D feature learning. While the use of RNNs and multiple viewpoints (typically $W=6$) increases computational complexity, the gains in feature quality are significant. Future work may involve optimizing viewpoint selection and scaling the approach to large-scale real-world LiDAR data from autonomous vehicles.

Summary updated

Abstract

Unsupervised feature learning for point clouds has been vital for large-scale point cloud understanding. Recent deep learning based methods depend on learning global geometry from self-reconstruction. However, these methods are still suffering from ineffective learning of local geometry, which significantly limits the discriminability of learned features. To resolve this issue, we propose MAP-VAE to enable the learning of global and local geometry by jointly leveraging global and local self-supervision. To enable effective local self-supervision, we introduce multi-angle analysis for point clouds. In a multi-angle scenario, we first split a point cloud into a front half and a back half from each angle, and then, train MAP-VAE to learn to predict a back half sequence from the corresponding front half sequence. MAP-VAE performs this half-to-half prediction using RNN to simultaneously learn each local geometry and the spatial relationship among them. In addition, MAP-VAE also learns global geometry via self-reconstruction, where we employ a variational constraint to facilitate novel shape generation. The outperforming results in four shape analysis tasks show that MAP-VAE can learn more discriminative global or local features than the state-of-the-art methods.
BibTeX
@inproceedings{han2019multi,
  author = {Zhizhong Han and Xiyang Wang and Yu-Shen Liu and Matthias Zwicker},
  title = {{Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.01054},
}
Figure