TL;DR
LOMA is a 3D semantic occupancy prediction network that fuses camera imagery with language priors derived from Vision-Language Models (VLMs). By projecting voxel features onto orthogonal 2D tri-planes and modeling them with linear-complexity State Space Models (Mamba), LOMA achieves global multi-modal context propagation without the heavy computational burden of traditional 3D attention.
What Problem Does This Paper Address?
In autonomous driving, vision-based 3D semantic scene completion (SSC) aims to predict both the occupancy state (free vs. occupied) and semantic label of every voxel in a 3D grid using camera images. While camera-only systems are cost-effective, they struggle with two major issues:
- Weak Geometric and Semantic Priors: 2D images lack explicit depth. While monocular depth estimation helps, vision-only cues struggle to infer occluded 3D geometry and distinguish fine-grained semantics simultaneously.
- The Quadratic Bottleneck of 3D Attention: Standard Transformer attention scales quadratically ($O(N^2)$) with the sequence length. In a dense 3D voxel grid (e.g., $256 \times 256 \times 32 \approx 2.1 \times 10^6$ voxels), global 3D attention is computationally prohibitive. Existing methods fall back on local or deformable attention, sacrificing long-range scene-level context.
Methods
LOMA addresses these challenges through a vision-language framework powered by state space sequence modeling:
[RGB Image + Text Prompts]
│
▼
[VL Scene Generator (VSG)] ──► Extracts 2D text/vision features & unprojects to 3D
│
▼
[3D Vision & Language Volumes]
│
▼
[Multi-Scale Tri-plane Fusion Mamba (MS-TFM)]
├─ Sparse 3D Convolutions (downsampling)
├─ Tri-plane Projection (XY, YZ, ZX planes)
├─ Shared 2D Mamba / SSM (global linear O(N) fusion)
└─ Linear 3D Unprojection + Dense Deconvolution
│
▼
[3D Semantic Occupancy Prediction]
- VL-aware Scene Generator (VSG): Leverages a pre-trained VLM (LSeg) to extract aligned 2D visual embeddings and text class embeddings. It assigns pixel-level semantic categories, which are then unprojected into 3D voxel space using camera calibration. This injects explicit semantic labels and implicit geometric scale priors (e.g., knowing an object is a "truck" immediately suggests rough bounding proportions in 3D).
- Tri-plane Fusion Mamba (TFM) Block: Instead of standard 3D self-attention, LOMA flattens the concatenated 3D vision and language features onto three orthogonal 2D planes ($XY, YZ, ZX$). A shared 2D State Space Model (SSM / Mamba block) processes these planes with $O(N)$ linear complexity, propagating features across modalities and from occupied voxels to empty ones. The features are then projected back to 3D and summed.
- Multi-Scale Extension (MS-TFM): Because the initial lifted 3D voxels are sparse, LOMA downsamples them using 3D sparse submanifold convolutions to save computation, applies TFM at multiple scales, and upsamples the resulting dense representations using regular 3D deconvolutions.
Datasets
The authors evaluate LOMA on two standard outdoor driving benchmarks:
- SemanticKITTI: 22 driving sequences (split 10/1/11 for train/val/test) annotated with 20 semantic categories over a $51.2\text{ m} \times 51.2\text{ m} \times 6.4\text{ m}$ volume ($256 \times 256 \times 32$ grid with $0.2\text{ m}$ voxels).
- SSCBench-KITTI360: Street-view dataset featuring 19 semantic classes, evaluated on a matching $256 \times 256 \times 32$ voxel grid.
No new datasets were introduced.
Key Insights and Ideas
- Language carries implicit 3D geometry: When a model identifies a voxel as belonging to a "car," human language carries a strong implicit prior about physical extents (e.g., roughly $4\text{m} \times 2\text{m} \times 2\text{m}$). The ablation studies confirm that adding VLM language embeddings improved geometric occupancy (IoU) significantly, not just semantic classification (mIoU).
- Mamba on Tri-planes breaks the 3D complexity curse: Standard 3D Transformers either blow up memory or restrict perception to local neighborhoods. Decomposing the volume into 2D tri-planes and applying visual Mamba provides full global receptive fields at a fraction of the compute (26.9G FLOPs for LOMA vs. 69.1G for 3D Swin Transformers).
- Sparse-to-Dense progression: Leveraging 3D sparse convolutions during downsampling respects the reality that most lifted camera voxels are initially empty. The global propagation of the TFM block fills in unobserved empty spaces, enabling dense deconvolutions for final decoding.
Key Results
- SemanticKITTI (Test): Achieves 15.10% mIoU (new state-of-the-art among camera-based methods, surpassing Symphonies at 15.04% and MonoOcc at 13.80%) and 43.01% IoU (second best, behind HASSC at 43.40%).
- SSCBench-KITTI360 (Test): Reaches 46.35% IoU (+2.23% over Symphonies' 44.12%) and 18.28% mIoU.
- Efficiency vs. Transformers: On SemanticKITTI validation, LOMA's SSM block outperforms 3D Swin Transformer (44.23 vs. 42.57 IoU; 14.81 vs. 13.73 mIoU) while cutting FLOPs by over $60\%$ (26.9G vs. 69.1G).
Limitations and Future Work
- Dependence on 2D Upstream Models: LOMA relies on frozen external models (MobileStereoNet for depth and LSeg for vision-language alignment); errors or misalignments in 2D predictions can propagate directly into the 3D voxel grid.
- Camera-Only Input: The current formulation relies exclusively on RGB cameras and text prompts.
- Future Directions: The authors plan to extend LOMA to multi-sensor configurations (e.g., integrating LiDAR or radar) to improve geometric accuracy under poor visibility or complex lighting conditions.
Summary updated