Abstract
Vision-based 3D occupancy prediction has become a popular research task due to its versatility and affordability. Conventional methods usually project image-based vision features to 3D space and learn geometric information through attention mechanisms. However, these works face two main challenges: limited geometric information in 2D images and local restricted interaction due to the quadratic complexity of attention. We propose LOMA, a language-assisted 3D semantic occupancy prediction network. LOMA introduces a VL-aware Scene Generator (VSG) to provide implicit geometric knowledge and explicit semantic information from language. Furthermore, a Tri-plane Fusion Mamba (TFM) block efficiently fuses 3D language and vision features with global modeling at low computational cost. Experiments on SemanticKITTI and SSCBench-KITTI360 show that LOMA achieves new state-of-the-art performance in both geometric and semantic completion tasks.
Introduction
3D Semantic Occupancy Prediction (SSC) is the task of predicting which parts of a 3D scene are occupied and identifying the objects within those spaces (e.g., cars, roads, vegetation) using 2D camera images. This capability is vital for autonomous vehicles to navigate safely. However, current methods face two major hurdles:
- Geometric Ambiguity: 2D images lack depth, making it inherently difficult to infer 3D volumes from flat pixels, especially in large-scale outdoor environments.
- The "Attention Tax": Most high-performing models rely on Transformers. While powerful, the attention mechanism's quadratic complexity often forces researchers to use "local" attention, which prevents the model from understanding the global structure of the scene.
Methodology
LOMA addresses these challenges through two primary innovations: the VL-aware Scene Generator (VSG) and the Tri-plane Fusion Mamba (TFM) block.
VL-aware Scene Generator (VSG)
Instead of relying solely on visual features, LOMA utilizes a pre-trained Vision-Language Model (specifically LSeg) to generate a 3D language feature. Language provides a geometric prior; for example, the word "car" evokes a specific 3D volume even if the camera only captures a portion of the vehicle. This module provides implicit geometric knowledge and explicit semantic labels to guide the 3D reconstruction.
Tri-plane Fusion Mamba (TFM)
To solve the computational bottleneck of Transformers, LOMA employs Mamba, a type of State Space Model. Mamba offers linear-time complexity, allowing for global modeling at a fraction of the cost. The model projects 3D features onto three orthogonal 2D planes (XY, YZ, ZX). A shared Mamba block processes these planes to capture global context, and the results are fused back into 3D space.
Experimental Results
The model was trained and evaluated on two industry-standard benchmarks: SemanticKITTI and SSCBench-KITTI360. LOMA achieved new state-of-the-art (SOTA) results on both datasets:
- SemanticKITTI: Achieved a 15.10 mIoU (mean Intersection over Union), outperforming previous top models like Symphonies (15.04) and HASSC (13.34).
- SSCBench-KITTI360: Achieved an IoU of 46.35 for occupancy and 18.28 mIoU for semantics, showing significant improvements over TPVFormer and VoxFormer.
Efficiency analyses showed that the Mamba-based TFM block achieved better performance than Swin3D or Deformable Attention while maintaining a competitive parameter count and lower FLOPs (Floating Point Operations).
Conclusion
LOMA demonstrates that semantic labels act as powerful "anchors" for geometry and that the linear scaling of State Space Models like Mamba is a superior architectural choice for dense 3D voxel grids. Future work may involve reducing dependency on specific pre-trained VLMs and incorporating additional sensor modalities like LiDAR or Radar to create a more robust multi-modal perception system.
Summary updated