NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMs

Nursena Köprücü, Destiny Okpekpe, Antonio Orvieto

Abstract

Transformers have become dominant in large-scale deep learning tasks across various domains, including text, 2D and 3D vision. However, the quadratic complexity of their attention mechanism limits their efficiency as the sequence length increases, particularly in high-resolution 3D data such as point clouds. Recently, state space models (SSMs) like Mamba have emerged as promising alternatives, offering linear complexity, scalability, and high performance in long-sequence tasks. The key challenge in the application of SSMs in this domain lies in reconciling the non-sequential structure of point clouds with the inherently directional (or bi-directional) order-dependent processing of recurrent models like Mamba. To achieve this, previous research proposed reorganizing point clouds along multiple directions or predetermined paths in 3D space, concatenating the results to produce a single 1D sequence capturing different views. In our work, we introduce a method to convert point clouds into 1D sequences that maintain 3D spatial structure with no need for data replication, allowing Mamba sequential processing to be applied effectively in an almost permutation-invariant manner. In contrast to other works, we found that our method does not require positional embeddings and allows for shorter sequence lengths while still achieving state-of-the-art results in ModelNet40 and ScanObjectNN datasets and surpassing Transformer-based models in both accuracy and efficiency.

Introduction: The Quadratic Bottleneck in 3D Vision

In the world of deep learning, Transformers are the reigning champions, but they suffer from a "quadratic problem": as the input sequence (or number of 3D points) grows, the computational cost explodes. State Space Models (SSMs), specifically the Mamba architecture, have emerged as a linear-time alternative that scales efficiently with long sequences.

However, applying Mamba to 3D point clouds—inherently unordered sets of $(x, y, z)$ coordinates—presents a significant challenge. Previous attempts, such as PointMamba, forced an order by sorting points along the X, Y, and Z axes and concatenating them. This "multi-scan" approach triples the sequence length ($3N$), introduces massive redundancy, and relies heavily on positional embeddings (PE) to maintain spatial context. NIMBA addresses these inefficiencies by introducing a principled way to process point clouds as spatially-aware 1D sequences.

Methods: Spatially-Aware Reordering

The core innovation of NIMBA is a spatially-aware 1D reordering strategy that preserves local proximity without data replication. The model utilizes Farthest Point Sampling (FPS) to select center points and k-Nearest Neighbors (kNN) to group nearby points into patches.

Instead of simple axis-based sorting, NIMBA employs a proximity check. It begins with a Y-axis sort but iteratively adjusts the sequence to ensure that consecutive points in the 1D list are physically close in 3D Euclidean space (within a threshold $r$). Surprisingly, the authors found that a well-ordered sequence provides sufficient implicit spatial information, making positional embeddings unnecessary. This removal actually improves the model's ability to generalize across rotations.

NIMBA's single-sequence approach vs. PointMamba's triple-scan sequence.

Key Results and Insights

NIMBA demonstrates that positional embeddings can often act as a "crutch" for models that fail to understand spatial relationships. By providing a principled sequence order, NIMBA achieves superior robustness to rotations and jittering compared to models relying on absolute coordinate-based PEs.

  • Efficiency Gains: By maintaining a sequence length of $N$ rather than $3N$, NIMBA reduces training time by 14-17% compared to multi-scan SSM approaches.
  • State-of-the-Art Performance: NIMBA outperformed both Transformer-based models (like Point-MAE) and previous Mamba models on the ScanObjectNN benchmark. Notably, on the challenging PB-T50-RS split, NIMBA achieved 84.21% accuracy.
Accuracy comparison across ModelNet and ScanObjectNN variants.

Conclusion

NIMBA proves that SSMs can be highly effective for 3D vision when the data is structured to respect its inherent spatial properties. While the model achieves state-of-the-art results, the authors noted that performance gains leveled off at very large parameter counts. Future work will explore optimizing SSM architectures specifically for the nuances of 3D data rather than directly porting NLP-centric designs. Additionally, the unexpected performance decline when using Mamba2 suggests that the theoretical interaction between SSMs and non-sequential data requires further investigation.

This summary is based on the research paper "NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMs".

Summary updated

Abstract

Transformers have become dominant in large-scale deep learning tasks across various domains, including text, 2D and 3D vision. However, the quadratic complexity of their attention mechanism limits their efficiency as the sequence length increases, particularly in high-resolution 3D data such as point clouds. Recently, state space models (SSMs) like Mamba have emerged as promising alternatives, offering linear complexity, scalability, and high performance in long-sequence tasks. The key challenge in the application of SSMs in this domain lies in reconciling the non-sequential structure of point clouds with the inherently directional (or bi-directional) order-dependent processing of recurrent models like Mamba. To achieve this, previous research proposed reorganizing point clouds along multiple directions or predetermined paths in 3D space, concatenating the results to produce a single 1D sequence capturing different views. In our work, we introduce a method to convert point clouds into 1D sequences that maintain 3D spatial structure with no need for data replication, allowing Mamba sequential processing to be applied effectively in an almost permutation-invariant manner. In contrast to other works, we found that our method does not require positional embeddings and allows for shorter sequence lengths while still achieving state-of-the-art results in ModelNet40 and ScanObjectNN datasets and surpassing Transformer-based models in both accuracy and efficiency.
BibTeX
@misc{koprucu2024nimba,
  author = {Nursena Köprücü and Destiny Okpekpe and Antonio Orvieto},
  title = {{NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMs}},
  year = {2024},
  eprint = {2411.00151},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00151},
}
Figure