LOMA: Language-assisted Semantic Occupancy Network via Triplane Mamba

Yubo Cui, Zhiheng Li, Jiaqiang Wang, Zheng Fang

Abstract

Vision-based 3D occupancy prediction has become a popular research task due to its versatility and affordability. Conventional methods usually project image-based vision features to 3D space and learn geometric information through attention mechanisms. However, these works face two main challenges: limited geometric information in 2D images and local restricted interaction due to the quadratic complexity of attention. We propose LOMA, a language-assisted 3D semantic occupancy prediction network. LOMA introduces a VL-aware Scene Generator (VSG) to provide implicit geometric knowledge and explicit semantic information from language. Furthermore, a Tri-plane Fusion Mamba (TFM) block efficiently fuses 3D language and vision features with global modeling at low computational cost. Experiments on SemanticKITTI and SSCBench-KITTI360 show that LOMA achieves new state-of-the-art performance in both geometric and semantic completion tasks.

Introduction

3D Semantic Occupancy Prediction (SSC) is the task of predicting which parts of a 3D scene are occupied and identifying the objects within those spaces (e.g., cars, roads, vegetation) using 2D camera images. This capability is vital for autonomous vehicles to navigate safely. However, current methods face two major hurdles:

  1. Geometric Ambiguity: 2D images lack depth, making it inherently difficult to infer 3D volumes from flat pixels, especially in large-scale outdoor environments.
  2. The "Attention Tax": Most high-performing models rely on Transformers. While powerful, the attention mechanism's quadratic complexity often forces researchers to use "local" attention, which prevents the model from understanding the global structure of the scene.

Methodology

LOMA addresses these challenges through two primary innovations: the VL-aware Scene Generator (VSG) and the Tri-plane Fusion Mamba (TFM) block.

VL-aware Scene Generator (VSG)

Instead of relying solely on visual features, LOMA utilizes a pre-trained Vision-Language Model (specifically LSeg) to generate a 3D language feature. Language provides a geometric prior; for example, the word "car" evokes a specific 3D volume even if the camera only captures a portion of the vehicle. This module provides implicit geometric knowledge and explicit semantic labels to guide the 3D reconstruction.

Tri-plane Fusion Mamba (TFM)

To solve the computational bottleneck of Transformers, LOMA employs Mamba, a type of State Space Model. Mamba offers linear-time complexity, allowing for global modeling at a fraction of the cost. The model projects 3D features onto three orthogonal 2D planes (XY, YZ, ZX). A shared Mamba block processes these planes to capture global context, and the results are fused back into 3D space.

The LOMA architecture featuring the VL-aware Scene Generator (VSG) and MS-TFM Layer.

Experimental Results

The model was trained and evaluated on two industry-standard benchmarks: SemanticKITTI and SSCBench-KITTI360. LOMA achieved new state-of-the-art (SOTA) results on both datasets:

  • SemanticKITTI: Achieved a 15.10 mIoU (mean Intersection over Union), outperforming previous top models like Symphonies (15.04) and HASSC (13.34).
  • SSCBench-KITTI360: Achieved an IoU of 46.35 for occupancy and 18.28 mIoU for semantics, showing significant improvements over TPVFormer and VoxFormer.
Performance comparison on the SemanticKITTI dataset showing LOMA's state-of-the-art results.

Efficiency analyses showed that the Mamba-based TFM block achieved better performance than Swin3D or Deformable Attention while maintaining a competitive parameter count and lower FLOPs (Floating Point Operations).

Conclusion

LOMA demonstrates that semantic labels act as powerful "anchors" for geometry and that the linear scaling of State Space Models like Mamba is a superior architectural choice for dense 3D voxel grids. Future work may involve reducing dependency on specific pre-trained VLMs and incorporating additional sensor modalities like LiDAR or Radar to create a more robust multi-modal perception system.

Summary updated

Abstract

Vision-based 3D occupancy prediction has become a popular research task due to its versatility and affordability. Nowadays, conventional methods usually project the image-based vision features to 3D space and learn the geometric information through the attention mechanism, enabling the 3D semantic occupancy prediction. However, these works usually face two main challenges: 1) Limited geometric information. Due to the lack of geometric information in the image itself, it is challenging to directly predict 3D space information, especially in large-scale outdoor scenes. 2) Local restricted interaction. Due to the quadratic complexity of the attention mechanism, they often use modified local attention to fuse features, resulting in a restricted fusion. To address these problems, in this paper, we propose a language-assisted 3D semantic occupancy prediction network, named LOMA. In the proposed vision-language framework, we first introduce a VL-aware Scene Generator (VSG) module to generate the 3D language feature of the scene. By leveraging the vision-language model, this module provides implicit geometric knowledge and explicit semantic information from the language. Furthermore, we present a Tri-plane Fusion Mamba (TFM) block to efficiently fuse the 3D language feature and 3D vision feature. The proposed module not only fuses the two features with global modeling but also avoids too much computation costs. Experiments on the SemanticKITTI and SSCBench-KITTI360 datasets show that our algorithm achieves new state-of-the-art performances in both geometric and semantic completion tasks. Our code will be open soon.
BibTeX
@inproceedings{cui2025loma,
  author = {Yubo Cui and Zhiheng Li and Jiaqiang Wang and Zheng Fang},
  title = {{LOMA: Language-assisted Semantic Occupancy Network via Triplane Mamba}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=5Yyi3Z3On1},
}
Figure