NIMBA : Towards Robust and Principled Processing of Point Clouds With SSMs

Nursena Koprucu, Destiny Okpekpe, Antonio OrvietoSubmission to ICLR 2025 · Rejected

TL;DR

Applying recurrent state space models (SSMs) like Mamba to unordered 3D point clouds usually requires hacky workarounds, such as tripling sequence lengths across coordinate axes and relying on positional embeddings. NIMBA introduces a simple, proximity-preserving serialization algorithm that orders 3D patches into a single 1D sequence without data replication or positional embeddings, boosting accuracy, rotation robustness, and training speed.

What Problem Does This Paper Address?

Transformers excel at 3D point cloud analysis because self-attention treats inputs as permutation-invariant sets. However, standard attention scales quadratically ($\mathcal{O}(N^2)$) with sequence length, creating computational bottlenecks for high-resolution 3D data.

State space models (SSMs) like Mamba offer linear complexity ($\mathcal{O}(N)$), but they are inherently sequential and order-sensitive. To force unordered 3D points into a sequential model, existing methods (e.g., PointMamba) replicate point sequences along the $x$, $y$, and $z$ axes—tripling the sequence length to $3N$—and lean heavily on explicit positional embeddings. This replication introduces redundancy, slows down training, and fails to guarantee that spatially close 3D points remain neighbors in the resulting 1D sequence.

Methods

The authors propose NIMBA (a portmanteau of Nimbus and Mamba), which eliminates sequence replication and removes positional embeddings by constructing a spatially coherent 1D sequence:

  1. Standard Patching: Point cloud centers are sampled via Farthest Point Sampling (FPS), and local patches are constructed around them using $k$-Nearest Neighbors ($k$-NN). Local patch embeddings are extracted using standard convolutions and linear layers.
  2. Initial Axis Ordering: Center coordinates are initially sorted along the $y$-axis (chosen because it empirically reduces search time in the next phase).
  3. Proximity-Check Reordering: NIMBA scans through the sequence and checks if the Euclidean distance between consecutive centers exceeds a threshold ($\|C_i - C_{i+1}\| < r$, with $r = 0.8$ for normalized $[-1, 1]^3$ coordinates). If two points are too far apart, the algorithm searches ahead for the nearest point within distance $r$ and moves it next in line.
  4. No Positional Embeddings: Because the serialization inherently preserves local geometry and Mamba's recurrence natively encodes order, NIMBA discards center/positional embeddings entirely.

Datasets

The authors evaluate NIMBA across standard benchmarks:

  • ModelNet40: Synthetic dataset of 12,311 clean CAD models across 40 categories used for classification.
  • ScanObjectNN: Real-world, noisy 3D indoor scene objects across 15 categories, evaluated on three benchmark splits: OBJ-ONLY, OBJ-BG (with background), and PB-T50-RS (perturbed with rotation and scaling).
  • ShapeNetPart: 31,693 CAD models across 16 categories used for 3D part segmentation.

No new datasets were introduced.

Key Insights and Ideas

  • Positional embeddings are redundant if sequence ordering is geometrically sound: In language modeling, SSMs do not need positional embeddings. In 3D vision, prior SSMs needed them primarily as a crutch because axis-wise sorting tore apart spatial neighborhoods. By preserving local distances during serialization, NIMBA can drop positional embeddings with minimal performance impact.
  • Natural robustness to 3D rotation: Standard axis-wise sorting is fragile to rotations because the coordinate projections shift completely. Because NIMBA relies on pairwise Euclidean distances between points (which are isometric under rigid rotations), it exhibits superior robustness against 3D rotations and noise.
  • Length $N$ beats $3N$: Replicating sequences along three separate axes is an inefficient way to provide multi-view context. A single, coherent sequence of length $N$ matches or outperforms $3N$ sequences while saving compute and memory.

Key Results

  • Classification Performance: On the challenging ScanObjectNN OBJ-BG split, NIMBA achieves 89.80% accuracy (at 23.86M parameters), outperforming PointMamba (88.01%) and Point-MAE (86.77%). On PB-T50-RS, NIMBA reaches 84.21% vs. PointMamba's 83.01%.
  • Efficiency: Processing sequences of length $N$ rather than $3N$ reduces training time over 300 epochs by ~14% on ModelNet40 and ~17% on ScanObjectNN.
  • Impact of Positional Embeddings (PE): Removing PE causes a performance collapse in baseline models (Point-MAE drops by $6.53\%$, PointMamba by $4.11\%$, and PoinTramba by $5.96\%$). NIMBA only drops by $1.68\%$—and its PE-free score (88.12%) still beats PointMamba with PE (87.80%).
  • Segmentation: On ShapeNetPart, NIMBA attains 84.36% class mIoU, outperforming PointMamba (83.37%) and Point-MAE (83.91%).

Limitations and Future Work

  • Diminishing returns from scaling: The authors note that scaling NIMBA from 12.3M to 23.86M parameters yielded only marginal accuracy gains, indicating that larger SSM backbones face optimization challenges in point clouds.
  • Issues with newer SSM variants: Replacing the base Mamba block with Hydra (a bidirectional SSM mixer based on Mamba2) caused an accuracy drop (~86.4%), suggesting that Mamba2-style architectures require specialized tuning for 3D tasks.
  • Hybrid integrations: Initial experiments integrating NIMBA into hybrid attention-SSM architectures encountered integration challenges, which the authors leave as an open area for future research.

Summary updated

Abstract

Transformers have become dominant in large-scale deep learning tasks across various domains, including text, 2D and 3D vision. However, the quadratic complexity of their attention mechanism limits their efficiency as the sequence length increases, particularly in high-resolution 3D data such as point clouds. Recently, state space models (SSMs) like Mamba have emerged as promising alternatives, offering linear complexity, scalability, and high performance in long-sequence tasks. The key challenge in the application of SSMs in this domain lies in reconciling the non-sequential structure of point clouds with the inherently directional (or bi-directional) order-dependent processing of recurrent models like Mamba. To achieve this, previous research proposed reorganizing point clouds along multiple directions or predetermined paths in 3D space, concatenating the results to produce a single 1D sequence capturing different views. In our work we introduce a method to convert point clouds into 1D sequences that maintains 3D spatial structure with no need for data replication, allowing Mamba’s sequential processing to be applied effectively in an almost permutation-invariant manner. In contrast to other works, we found that our method does not require positional embeddings, and allows for shorter sequence lengths while still achieving state-of-the-art results in ModelNet40 and ScanObjectNN datasets and surpassing Transformer-based models in both accuracy and efficiency.
BibTeX
@misc{koprucu2024nimba,
  author = {Nursena Koprucu and Destiny Okpekpe and Antonio Orvieto},
  title = {{NIMBA : Towards Robust and Principled Processing of Point Clouds With SSMs}},
  year = {2024},
  eprint = {2411.00151},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00151},
}
Figure