Abstract
Recently, Spiking Neural Networks (SNNs), enjoying extreme energy efficiency, have drawn much research attention on 2D visual recognition and shown gradually increasing application potential. However, it still remains underexplored whether SNNs can be generalized to 3D recognition. To this end, we present Spiking PointNet in the paper, the first spiking neural model for efficient deep learning on point clouds. We discover that the two huge obstacles limiting the application of SNNs in point clouds are: the intrinsic optimization obstacle of SNNs that impedes the training of a big spiking model with large time steps, and the expensive memory and computation cost of PointNet that makes training a big spiking point model unrealistic. To solve the problems simultaneously, we present a trained-less but learning-more paradigm for Spiking PointNet with theoretical justifications and in-depth experimental analysis. In specific, our Spiking PointNet is trained with only a single time step but can obtain better performance with multiple time steps inference, compared to the one trained directly with multiple time steps. We conduct various experiments on ModelNet10, ModelNet40 to demonstrate the effectiveness of Spiking PointNet. Notably, our Spiking PointNet even can outperform its ANN counterpart, which is rare in the SNN field thus providing a potential research direction for the following work. Moreover, Spiking PointNet shows impressive speedup and storage saving in the training phase.
Introduction
Deep learning on 3D point clouds, such as those generated by LiDAR, is computationally intensive and energy-demanding, posing significant challenges for deployment on edge devices like autonomous drones or AR glasses. Spiking Neural Networks (SNNs) offer a biologically inspired alternative, communicating via discrete "spikes." This sparsity allows for massive energy savings through addition-only computations.
However, generalizing SNNs to 3D point clouds faces two primary hurdles:
- The Optimization Wall: SNNs utilize non-differentiable firing functions. While surrogate gradients mitigate this, training large models over multiple time steps often results in vanishing or exploding gradients.
- The Memory Wall: Architectures like PointNet are inherently memory-intensive. Multiplying this by several temporal time steps—a standard requirement for SNNs—renders training on current hardware nearly impossible.
Methods: The "Trained-less but Learning-more" Paradigm
To address these challenges, the authors propose Spiking PointNet, which adapts the classic PointNet architecture by replacing standard ReLU neurons with Leaky Integrate-and-Fire (LIF) spiking neurons. The core innovation lies in a novel training strategy:
- Single-Step Training ($T=1$): Instead of training over many time steps, the model is trained using only a single time step. This bypasses the gradient instability and memory bloat associated with long-sequence training.
- Multi-Step Inference: During deployment, the model is run for multiple time steps. This allows the network to benefit from temporal integration and residual membrane potentials, effectively acting as an ensemble of slightly different "views" of the data.
- Membrane Potential Perturbation (MPP): To improve generalization across varying initial states during multi-step inference, the authors introduce random noise to the initial membrane potential during training, serving as a regularizer.
Gradient Optimization
By restricting training to $T=1$, the authors can utilize a larger coefficient ($k$) for surrogate gradients. This leads to more accurate weight updates without the risk of gradient explosion typically encountered in multi-step SNN training.
Experimental Results
The model was evaluated on the ModelNet10 and ModelNet40 benchmarks.
- Accuracy: Spiking PointNet achieved 93.31% on ModelNet10 and 88.61% on ModelNet40. Remarkably, on ModelNet10, it outperformed the original ANN-based PointNet (92.98%).
- Energy Efficiency: The SNN architecture demonstrated a 15.2-fold reduction in energy consumption compared to its ANN counterpart, as it replaces floating-point multiplications with simple additions.
- Training Efficiency: The single-step training paradigm significantly reduced memory requirements and training time compared to traditional multi-step SNN approaches.
Conclusion
Spiking PointNet represents a significant step forward in bringing energy-efficient spiking neurons to 3D vision. By overcoming optimization and memory constraints through the "trained-less but learning-more" paradigm, it achieves state-of-the-art performance with a fraction of the energy cost. Future work will explore applying these principles to dynamic spatio-temporal data and more complex 3D architectures like PointNet++ or Transformers.
Summary updated