Abstract
Deep generative architectures provide a way to model not only images but also complex, 3-dimensional objects, such as point clouds. In this work, we present a novel method to obtain meaningful representations of 3D shapes that can be used for challenging tasks including 3D points generation, reconstruction, compression, and clustering. Contrary to existing methods for 3D point cloud generation that train separate decoupled models for representation learning and generation, our approach is the first end-to-end solution that allows to simultaneously learn a latent space of representation and generate 3D shape out of it. Moreover, our model is capable of learning meaningful compact binary descriptors with adversarial training conducted on a latent space. To achieve this goal, we extend a deep Adversarial Autoencoder model (AAE) to accept 3D input and create 3D output. Thanks to our end-to-end training regime, the resulting method called 3D Adversarial Autoencoder (3dAAE) obtains either binary or continuous latent space that covers a much wider portion of training data distribution. Finally, our quantitative evaluation shows that 3dAAE provides state-of-the-art results for 3D points clustering and 3D object retrieval.
Introduction
As LiDAR and RGB-D sensors become ubiquitous in autonomous vehicles and robotics, the volume of 3D "point cloud" data is increasing exponentially. Processing this data in its raw form is computationally expensive and storage-intensive. Researchers face two primary hurdles: efficient representation learning (compressing complex shapes into compact "codes") and the limitations of decoupled training, where representation learning and generation are handled by separate models. This often leads to a "gappy" latent space that fails to produce realistic results during interpolation.
Methodology: The 3D Adversarial Autoencoder (3dAAE)
The authors propose 3dAAE, an end-to-end solution that simultaneously learns a latent representation and a generative model. The architecture consists of three components:
- Encoder: A permutation-invariant network based on PointNet that maps 3D input into a latent vector $z$.
- Generator (Decoder): A fully connected network that reconstructs the 3D point cloud from $z$.
- Discriminator: An adversarial component that regularizes the latent space by distinguishing between encoded latent codes and samples from a target prior distribution.
Key Innovations
- Flexible Priors: Unlike standard Variational Autoencoders (VAEs), 3dAAE can utilize diverse priors (e.g., Gaussian Mixtures or Beta distributions) without requiring closed-form solutions for the distribution gap.
- Earth Mover’s Distance (EMD): By using EMD as the reconstruction loss, the model enforces a strict one-to-one mapping between points, resulting in significantly higher visual quality compared to the standard Chamfer Distance.
- Binary Descriptors: By employing a Beta distribution prior, the model can learn compact 100-bit binary descriptors that capture discriminative features efficiently.
Experimental Results
The model was evaluated on the ShapeNet and ModelNet40 datasets, using point clouds of 2,048 points each.
Generation and Reconstruction
3dAAE achieved state-of-the-art results in generation quality, recording a Jensen-Shannon Divergence (JSD) of 0.014 on the chair category. The adversarial regularization ensures a "dense" latent space, allowing for smooth interpolation between shapes and "latent arithmetic"—the ability to add or subtract specific geometric features (e.g., adding "rocker" features to a standard chair).
Classification and Retrieval
On the ModelNet40 benchmark, the model achieved 84.7% accuracy using continuous embeddings. Remarkably, even when compressed to 100-bit binary descriptors, it maintained nearly 80% accuracy, outperforming many continuous representations in retrieval tasks.
Conclusion and Future Work
3dAAE provides a robust framework for learning compact, meaningful representations of 3D data. While the use of Earth Mover’s Distance is computationally intensive, the resulting high-fidelity reconstructions and efficient binary descriptors mark a significant step forward. Future research will address scaling the model to denser point clouds and improving its robustness to the noisy, incomplete data typical of real-world LiDAR sensors.
Summary updated