L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention

Xinhai Liu, Zhizhong Han, Xin Wen, Yu-Shen Liu, Matthias Zwicker

Abstract

Point cloud understanding is a critical task in 3D computer vision, yet the unstructured nature of point sets poses significant challenges for traditional deep learning architectures. We introduce the L2G Auto-encoder (L2G-AE), an unsupervised framework designed to learn robust geometric representations through a local-to-global reconstruction task. By employing a hierarchical self-attention mechanism in the encoder and an RNN-based progressive decoder, the model effectively captures multi-scale local features and their global relationships. L2G-AE achieves state-of-the-art performance on unsupervised classification and shape retrieval benchmarks, demonstrating the efficacy of bottom-up reconstruction for 3D shape understanding.

Introduction

Point clouds, typically represented as sets of $(x, y, z)$ coordinates, are the primary output of 3D sensors like LiDAR. Unlike 2D images, their lack of a regular grid structure makes standard convolutional operations inapplicable. While architectures like PointNet have advanced the field, many existing Auto-encoders (AEs) focus predominantly on global structure, often neglecting the subtle local geometric features essential for fine-grained shape understanding. To address this, we propose a "Local-to-Global" (L2G) approach that emphasizes bottom-up reconstruction, forcing the model to understand how local patches compose the global form.

Methods

The L2G-AE architecture consists of a hierarchical self-attention encoder and a progressive RNN-based decoder, connected by a global feature vector.

Overview of the L2G Auto-encoder architecture, showing the hierarchical self-attention encoder and the progressive decoder.

Hierarchical Self-Attention Encoder

The encoder moves beyond simple max-pooling by employing a three-level attention mechanism to dynamically weight geometric features:

  1. Multi-scale Establishment: Centroids are sampled across the shape, and neighboring points are grouped into four distinct scales (neighborhood sizes).
  2. Point-level Attention: Captures relationships between points within a single patch.
  3. Scale-level Attention: Dynamically weights the contributions of different neighborhood sizes.
  4. Region-level Attention: Models the spatial relationships between different parts of the object.

Progressive RNN-based Decoder

The decoder reconstructs the shape by treating multi-scale local areas as a sequence, allowing the model to learn spatial relationships between a point and its increasingly larger neighborhoods.

The RNN-based local reconstruction process, illustrating how local region features are decoded into point sets and compared via Chamfer loss.
  1. Interpolation: The 1024-dimensional global feature is interpolated back to local region centroids using distance-based weighting.
  2. RNN Decoding: An LSTM-based RNN processes the multi-scale features as a sequence, learning the evolution of geometry from local patches to larger regions.
  3. Joint Loss: Training is guided by a combination of local and global reconstruction losses, calculated using the Chamfer Distance metric to compare point set similarity.

Results and Discussion

L2G-AE was evaluated on standard benchmarks including ModelNet10, ModelNet40, and ShapeNet55.

  • Unsupervised Classification: On ModelNet40, L2G-AE achieved 90.64% accuracy in an unsupervised setting. This significantly outperforms previous unsupervised leaders like FoldingNet (88.4%) and even exceeds the performance of the original supervised PointNet (89.2%).
  • Shape Retrieval: In retrieval tasks, L2G-AE achieved a mean Average Precision (mAP) of 67.81%, a substantial improvement over the previous state-of-the-art of 53.42%.
  • Point Cloud Upsampling: The model demonstrated the ability to take sparse inputs (625 points) and reconstruct dense, smooth outputs (10,000 points), performing competitively with specialized supervised upsampling networks like PU-Net.

The success of L2G-AE highlights that self-attention allows for a "soft" selection of features, capturing correlations that standard max-pooling ignores. Furthermore, treating geometry as a sequence for RNN decoding provides a powerful inductive bias for understanding spatial hierarchies.

Conclusion

The L2G Auto-encoder provides a robust unsupervised framework for point cloud representation learning. By focusing on local-to-global reconstruction, it captures rich geometric features that generalize well to various downstream tasks. Future work will explore scaling this approach to complex 3D scenes and optimizing the computational overhead associated with hierarchical attention and RNNs.

Summary updated

BibTeX
@inproceedings{liu2019l,
  author = {Xinhai Liu and Zhizhong Han and Xin Wen and Yu-Shen Liu and Matthias Zwicker},
  title = {{L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3350960},
}
Figure