FEAST-Mamba: FEAture and SpaTial Aware Mamba Network with Bidirectional Orthogonal Fusion for Cross-Modal Point Cloud Segmentation
Introduction
Point cloud segmentation—the task of labeling every point in a 3D scene—is a cornerstone of autonomous driving and augmented reality. While LiDAR provides precise spatial data, it lacks the rich texture and color of RGB images. Fusing these modalities is the logical solution, but it faces two major challenges: information redundancy and computational complexity. Traditional Transformers offer long-range context but suffer from quadratic complexity ($O(N^2)$). State Space Models (Mamba) provide a linear-time ($O(N)$) alternative but are highly sensitive to the order of input sequences, which is problematic for inherently disordered point clouds.
Methods
FEAST-Mamba introduces two primary innovations to address these issues: Bidirectional Orthogonal Fusion (BOF) and a novel reordering mechanism.
Bidirectional Orthogonal Fusion (BOF)
To minimize redundancy, BOF employs a two-step process. First, Bidirectional Feature Enhancement (BFE) uses cross-modal attention to allow 3D voxels and 2D pixels to enhance each other's features. Second, an Orthogonal Constraint is applied via a loss term ($L_{reg}$), forcing the 2D and 3D features to be mathematically independent. This ensures the model captures unique, complementary information from each sensor rather than redundant data.
Feature and Spatial Aware Reordering
Standard Mamba-based methods often rely on simple spatial sorting to linearize point clouds, which ignores semantic relationships.
FEAST-Mamba replaces this with a Dual-aware Ordering Sequence:
- Spatial-aware: Uses K-Nearest Neighbors (KNN) to group physically close points.
- Feature-aware: Uses cosine similarity to group points that are semantically similar in high-dimensional space, even if they are physically distant (e.g., different parts of the same object).
Key Results
FEAST-Mamba achieves state-of-the-art results across several benchmarks, including ScanNet (77.8% mIoU), S3DIS (74.7% mIoU), and SemanticKITTI (73.2% mIoU). Notably, it matches or exceeds the performance of heavyweights like Point Transformer v3 (PTv3) while maintaining linear complexity and higher inference speeds.
Qualitative analysis shows that FEAST-Mamba is particularly effective at identifying small or thin objects, such as fences and vegetation, which are often misclassified by models relying solely on spatial proximity.
Conclusion
By combining orthogonality-driven fusion with semantic-aware reordering, FEAST-Mamba demonstrates that high-performance 3D segmentation does not require the heavy computational cost of Transformers. Future work will focus on scaling this architecture to even larger scenes and exploring its application in 3D object detection and panoptic segmentation.
Summary updated