TGP: Two-modal occupancy prediction with 3D Gaussian and sparse points for 3D Environment Awareness

Chen Mu, Wenyu Chen, Mingchuan Yang, Yuan Zhang, Tao Han, Xinchi Li, Yunlong Li, Huaici Zhao

Abstract

3D semantic occupancy has rapidly become a research focus in the fields of robotics and autonomous driving environment perception due to its ability to provide more realistic geometric perception and its closer integration with downstream tasks. By performing occupancy prediction of the 3D space in the environment, the ability and robustness of scene understanding can be effectively improved. However, existing occupancy prediction tasks are primarily modeled using voxel or point cloud-based approaches: voxel-based network structures often suffer from the loss of spatial information due to the voxelization process, while point cloud-based methods, although better at retaining spatial location information, face limitations in representing volumetric structural details. To address this issue, we propose a dual-modal prediction method based on 3D Gaussian sets and sparse points, which balances both spatial location and volumetric structural information, achieving higher accuracy in semantic occupancy prediction. Specifically, our method adopts a Transformer-based architecture, taking 3D Gaussian sets, sparse points, and queries as inputs. Through the multi-layer structure of the Transformer, the enhanced queries and 3D Gaussian sets jointly contribute to the semantic occupancy prediction, and an adaptive fusion mechanism integrates the semantic outputs of both modalities to generate the final prediction results. Additionally, to further improve accuracy, we dynamically refine the point cloud at each layer, allowing for more precise location information during occupancy prediction. We conducted experiments on the Occ3DnuScenes dataset, and the experimental results demonstrate superior performance of the proposed method on IoU based metrics.

Introduction

In autonomous driving, "seeing" the world in 3D is moving beyond simple bounding boxes toward 3D Semantic Occupancy Prediction—predicting which parts of 3D space are occupied and by what (e.g., car, sidewalk, vegetation). Current approaches generally fall into two camps: voxel-based methods, which are computationally expensive and lose detail due to voxelization, and point-based methods, which retain precision but struggle to represent volumetric structure.

The authors of TGP identify a "missing middle": how to maintain the efficiency and precision of points while gaining the ability to represent the actual volume of irregular objects. TGP (Two-modal Gaussian and Points) addresses this by combining the precision of sparse point clouds with the volumetric modeling power of 3D Gaussians.

Methodology

The core innovation is a dual-modal decoder that processes two distinct representations of the 3D scene simultaneously within a Transformer-based architecture.

Overall architecture of TGP, showing the multi-layer dual-modal decoder structure that integrates sparse points, 3D Gaussians, and image-based queries.
  • Sparse Points: These act as precise anchors. The model uses a "coarse-to-fine" strategy, progressively splitting and refining point positions across six decoder layers to home in on object boundaries.
  • 3D Gaussians: Inspired by 3D Gaussian Splatting, each Gaussian carries attributes like position, volume (scale), rotation, and semantics. Unlike points, Gaussians can "stretch" and "rotate" to fill the actual volume of an object.
  • Transformer-based Fusion: Learnable queries extract features from multi-view camera images to update both point positions and Gaussian attributes.
  • Adaptive Semantic Fusion: A "Gaussian-to-Voxel Splatting" module aggregates local volumetric information, which is then fused with point-level semantic predictions using a sigmoid-weighted mechanism.

Experiments and Results

The researchers evaluated TGP on the Occ3D-nuScenes dataset, using standard mIoU and RayIoU metrics. TGP achieved superior performance across all major metrics:

  • Accuracy: The "Small" version (TGP-S) reached 34.5 mIoU and 39.4 RayIoU, outperforming previous state-of-the-art point-based models like OPUS-S (34.2 mIoU).
  • Qualitative Superiority: Visualizations demonstrate that TGP is significantly better at capturing the continuous volume of objects compared to point-based methods, which often appear "holey" or disconnected.
Qualitative comparison of occupancy prediction results between OPUS, TGP, and Ground-truth. TGP shows better volumetric continuity in areas like roadside vegetation and vehicle bodies.
  • Efficiency Trade-offs: While TGP is more accurate, it is slightly slower than pure point-based methods. TGP-S runs at 15.1 FPS, compared to OPUS-S at 20.7 FPS, representing a calculated trade-off between speed and geometric detail.

Key Insights

  • Complementary Modalities: Points excel at localization ("where"), while Gaussians excel at shape representation ("what shape"). Combining them allows the model to fill gaps that point-only models miss.
  • Shared Initialization: A key finding was that 3D Gaussians and sparse points must share the same initial positions to avoid a "distribution mismatch" that degrades performance.

Conclusion

TGP effectively bridges the gap between point-based and volumetric occupancy prediction. While the primary limitation is the added computational cost of processing 3D Gaussians, the model remains real-time capable at 15 FPS. Future work will focus on optimizing the inference speed of the Gaussian-to-voxel splatting process and exploring integration into downstream tasks like path planning.

Summary updated

Abstract

3D semantic occupancy has rapidly become a research focus in the fields of robotics and autonomous driving environment perception due to its ability to provide more realistic geometric perception and its closer integration with downstream tasks. By performing occupancy prediction of the 3D space in the environment, the ability and robustness of scene understanding can be effectively improved. However, existing occupancy prediction tasks are primarily modeled using voxel or point cloud-based approaches: voxel-based network structures often suffer from the loss of spatial information due to the voxelization process, while point cloud-based methods, although better at retaining spatial location information, face limitations in representing volumetric structural details. To address this issue, we propose a dual-modal prediction method based on 3D Gaussian sets and sparse points, which balances both spatial location and volumetric structural information, achieving higher accuracy in semantic occupancy prediction. Specifically, our method adopts a Transformer-based architecture, taking 3D Gaussian sets, sparse points, and queries as inputs. Through the multi-layer structure of the Transformer, the enhanced queries and 3D Gaussian sets jointly contribute to the semantic occupancy prediction, and an adaptive fusion mechanism integrates the semantic outputs of both modalities to generate the final prediction results. Additionally, to further improve accuracy, we dynamically refine the point cloud at each layer, allowing for more precise location information during occupancy prediction. We conducted experiments on the Occ3DnuScenes dataset, and the experimental results demonstrate superior performance of the proposed method on IoU based metrics.
BibTeX
@misc{mu2025tgp,
  author = {Chen Mu and Wenyu Chen and Mingchuan Yang and Yuan Zhang and Tao Han and Xinchi Li and Yunlong Li and Huaici Zhao},
  title = {{TGP: Two-modal occupancy prediction with 3D Gaussian and sparse points for 3D Environment Awareness}},
  year = {2025},
  eprint = {2503.09941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.09941},
}
Figure